In this article, we will explain how to resolve the java.io.FileNotFoundException: /home/log4j.properties in the Hadoop clusters.
Error:
og4j:ERROR Could not read configuration file from URL [file:/home//log4j.properties]. java.io.FileNotFoundException: /home/log4j.properties (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66) at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72) at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45) at org.apache.spark.internal.Logging$.org$apache$spark$internal$Logging$$isLog4j12(Logging.scala:217) at org.apache.spark.internal.Logging$class.initializeLogging(Logging.scala:122) at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:111) at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:71) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:79) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) log4j:ERROR Ignoring configuration file [file:/home//log4j.properties]. 22/08/20 13:34:42 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. 'spark.executor.memoryOverhead' instead. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 22/08/20 13:34:42 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. 'spark.executor.memoryOverhead' instead. 22/08/20 13:34:42 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. 'spark.executor.memoryOverhead' instead. 22/08/20 13:34:43 WARN HiveConf: HiveConf of name hive.masking.algo does not exist 22/08/20 13:34:43 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. 'spark.executor.memoryOverhead' instead. 22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043. 22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044. 22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045. 22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
Solution :
This type of issue belongs to performance related, we need to increase the JVM heap size and re-trigger the spark application or else we need to do some performance tuning on the scripts as Spark code standards with transactional or non transactional related thing as well.