In this article, we will explain how to resolve the java.io.FileNotFoundException: /home/log4j.properties in the Hadoop clusters.

Error:

og4j:ERROR Could not read configuration file from URL [file:/home//log4j.properties].

java.io.FileNotFoundException: /home/log4j.properties (No such file or directory)

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

at java.io.FileInputStream.<init>(FileInputStream.java:93)

at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)

at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)

at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)

at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)

at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)

at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)

at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)

at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)

at org.apache.spark.internal.Logging$.org$apache$spark$internal$Logging$$isLog4j12(Logging.scala:217)

at org.apache.spark.internal.Logging$class.initializeLogging(Logging.scala:122)

at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:111)

at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:71)

at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:79)

at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

log4j:ERROR Ignoring configuration file [file:/home//log4j.properties].

22/08/20 13:34:42 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future.    'spark.executor.memoryOverhead' instead.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

22/08/20 13:34:42 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future.    'spark.executor.memoryOverhead' instead.

22/08/20 13:34:42 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future.    'spark.executor.memoryOverhead' instead.

22/08/20 13:34:43 WARN HiveConf: HiveConf of name hive.masking.algo does not exist

22/08/20 13:34:43 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future.    'spark.executor.memoryOverhead' instead.

22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.

22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.

22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.

22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045.

22/08/20 13:34:44 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.

Solution :

This type of issue belongs to performance related, we need to increase the  JVM heap size  and re-trigger the spark application or else we need  to do some performance tuning on the scripts as Spark code standards with transactional or non transactional related thing as well.