[Resolved] How to resolve the Impala query throwing memory errors | Big Data | Hadoop

In this article, we will explain how to resolve the impala query memory issue:



ERROR:

 Basically, cannot perform hash join at edge node with id (4/5). Repartitioning did not reduce the size of a overturn partition. Repartitioning level 1. Number of rows 157214: PartitionedHashJoinNode (id=4 op=0 state=RepartitioningBuild #overturn_partitions=0) PhjBuilder: Hash partitions: 16: Hash partition 0 : partition code7ef90ad02aa0 Closed Hash partition 1 : partition code7ef5fbd94800 Closed Hash partition 2 : partition code7ef90ad021c0 Closed Hash partition 3 : partition code7eb6f9730c80 Closed Hash partition 4 : partition code7ef6854b2ce0 Closed Hash partition 5 : partition code7ef279728140 Closed Hash partition 6 : partition code7ef68e81d6a0 Closed Hash partition 7 : partition code7eac0856e7c0 Closed Hash partition 8 : partition code7eb2fea2fe40 Closed Hash partition 9 : partition code7eae47c497e0 Closed Hash partition 10 : partition code7ef4ce80b560 Closed Hash partition 11 : partition code7eaf7a934200 Closed Hash partition 12 : partition code685d9520 Closed Hash partition 13 : partition code7ef9f4f2a280 Closed Hash partition 14 : partition code7eee5d598e80 Closed Hash partition 15 : partition code7ef5ffc6ae00 overturn Build Rows: 157214 (Bytes pinned: 0) Probe hash partitions: 0: InputPartition: 0x1e88f6d0 Build Partition Closed overturn Probe Rows: 1 BufferPool::Client 0x7ebe0c7ceaa0 internal state: {BufferPool::Client 0x9eac94f25d40 name: HASH_JOIN_NODE id=4 partition code7ebe0c7ce900 write_status: buffers allocated 0 num_pages: 42 pinned_bytes: 2097152 dirty_unpinned_bytes: 0 in_flight_write_bytes: 2097152 reservation: {: reservation_limit 9223372036854775807 reservation 35651584 used_reservation 2097152 child_reservations 2097152 parent: : reservation_limit 9223372036854775807 reservation 17108566016 used_reservation 0 child_reservations 17108566016 parent: : reservation_limit 17136869184 reservation 17136869184 used_reservation 0 child_reservations 17136869184 parent: : reservation_limit 365072220160 reservation 220350939136 used_reservation 0 child_reservations 220350939136 parent: NULL} 1 pinned pages: BufferPool::Page 0x7ef9cc6de460 len: 2097152 pin_count: 1 buf: BufferPool::BufferHandle 0x7ef9cc6de4d8 client: 0x7ebe0c7ceaa0/0x9eac94f25d40 data: 0x7eaaf0c00000 len: 2097152 0 dirty unpinned pages: 1 in flight write pages: BufferPool::Page 0x7ef91dd36760 len: 2097152 pin_count: 0 buf: BufferPool::BufferHandle 0x7ef91dd367d8 client: 0x7ebe0c7ceaa0/0x9eac94f25d40 data: 0x7ec4a4200000 len: 2097152 }

Solution

When I triggered one of my Impala job got failed due to above error belongs to Impala memory related.




Previously, I used MEM_LIMIT = 7g

Please try to do below memory configuration in the script, try to increase memory limit:

set MEM_LIMIT = 8g

After I’ve changed the above configurations and re-trigger the jobs. It’s running fine. Anyone comment incase any other resolution find out for this type of issues. This type cases belongs to performance tuning related for this scenario. Most of the cases we need to increase memory related thing only.