Error : java.lang.OutOfMemoryError: Java heap space in Azure blob | Big Data | Azure

In this article, we will explain java.lang.OutOfMemoryError: Java heap space in Azure blob file system store for Big Data & Cloud developers.



Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space in Azure Blob

 

Exception message 'ERROR [HY000] [Microsoft][Hardy] (35) Error from server: error code: '2' error message: 'Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_, diagnostics=[Task failed, taskId=task_, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:243)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:375)
at org.apache.hadoop.io.ElasticByteBufferPool.getBuffer(ElasticByteBufferPool.java:596)
at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<init>(AbfsOutputStream.java:797)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createFile(AzureBlobFileSystemStore.java:118)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:287)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.jav...'.

The above error belongs to memory related. It seems like either Java heap memory or Outofmemory in the job. Here we provided simple resolution for this error.

Solution:

1.First, we need to check in Resource Manager. Whether any job is log run or consuming more containers.




2.If it is there need to kill the job from Resource Manger using putty.

Here we need to check the job status first:

yarn application -status application_id

Then need to kill the job using below command:

yarn application -kill application_id

3.After that please try to re-run from Azure Data Factory for the particular job.

I think this issue has been resolved, if it is still facing the same error. We need it figured out for RCA (Root Cause Analysis).

The above resolution is very simple to resolve the issue as per OutofMemory error.




Basically, memory related  issues are quite common  for Big Data clusters or Azure clusters.

Leave a Reply

Your email address will not be published. Required fields are marked *