First, I am writing the Spark application program on Scala Spark-shell in Scala program.
ClassNotFoundExceptions in Spark:
import org.apache.spark.SparkContext import org.apache.spark.SparkConf import org.apache.sql.SparkSession object sparkDemo{ def main(args:Array[String]): Unit={ val conf = new SparkCoonf().setAppName("Deply").setMaster("local[*])") val sc = nes SparkContext(conf) sc.setLogLevel("ERROR") val data = sc.textFile("file:///home//Spark//Test//Input.log") val file_data = data.filter(x=>x.contains("Population")) println("Filtered Data") println(fil_data.foreach(println)) fil_data.saveAsTextFile("file://home/Spark//Output") } }
While submitting the spark application jar file on spark-shell getting below error:
Scala> spark - submit --class sparkDemo.sparkDemo --master local[*] file:///home//Spark//Output/sparkDemo-0.0.1.-SNAPSHOT.jar
Error 1:
Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: Caused by: java.lang.ClassNotFoundException: kafka.DefaultSource
Solution:
bin\spark-submit --class sparkDemo.Main --master local[*] file:///home//Spark//Output/sparkDemo-0.0.1.-SNAPSHOT.jar
After submitting the Spark jar using the above command, it’s worked.
Error 2:
java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.avro.AvroFileFormat. Please find parameters.html at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:438) at org.apache.spark.sql.DaataFrameWriter.save(DataFrameWriter.scala:244) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:288) Caused by: java.lang.ClassNotFound Exception: org.apache.spark.sql.avro.AvroFileFormat.DefaultSource at scala.reflect.internal.util.AbstractFileClassLoader. findClass(AbstractFileClassLoader.scala:62)
Solution:
The above error belongs to the Avro file format exception due to incompatible issues with Spark. So first check with Spark Core and Spark SQL compatible.
After the fixed, the compatible issue for Spark Core & Spark SQL then go with Spark-Avro
Summary: Here are we resolved two errors like ClassNotFoundException one is Spark jar file submission and another one is Spark – Avro file exception. We provide exact solutions.