In Spark different types of cores are available for submit the Spark jar file.
- Hadoop Cluster Image:
spark-submit --master local[*] --num-executors 2 --driver-memory 512m --executor-memory 512m --executor-cores 1 --class wordcount SPARK-WC-WPRKSPACE.jar file:///home/Sreekanth/Spark/SPARK-SBT/Data.log file:///home/Sreekanth/SPARK-SBT/SPWCOUT90
- SPARK with SCALA Image:
Go to Spark jar file available path then run the below command:
$pwd spark-submit --class SparkWordCount SPARK - WC.jar file:///home/Sreekanth/Data.log file:///home/Sreekanth/PRAC/SPARK_OUTPUT
- To Run the Spark Job in Local Mode:
spark-submit --class com.spark.core.LinesWithErrorStarting --master local LINES - WITH - STARTING - SPARK_WC.jar. Data.log SparkOutDir
spark-submit --class com.spark.core.LinesWithErrorStarting --master local [5]LINES - WITH - STARTING - SPARK_WC.jar. Data.log SparkOutDir
spark-submit --class com.spark.core.LinesWithErrorStarting --master local [*] LINES - WITH - STARTING - SPARK_WC.jar. Data.log SparkOutDir
Note : local [*] indicates that the number of CPU CORES
- To Run the Spark Job in Cluster Mode:
spark-submit --class com.spark.core.LinesWithErrorStarting --master spark://localhost:7077 --deploy mode client LINES - WITH - STARTING - SPARK_WC.jar. Data.log SparkOutDir2
Note: The above command to run, we need not to start SPARK Daemons because in yarn client mode your driver program is running on the yarn client where you type the command to submit the spark application(may not be machine in the yarn cluster) In this mode the driver program is running on the client machine, the tasks are executed on the executors in the node managers of the YARN cluster.