Configuring YARN Container Memory Limits: Migration Challenges and Solutions from Hadoop v1 to v2

Introduction

In distributed computing frameworks, resource management is crucial for efficient task execution. The upgrade from Apache Hadoop v1 to v2 (introducing YARN as the resource manager) offers more flexible resource configuration but also presents new challenges. This article analyzes a typical case of container memory limit issues during migration and provides solutions based on YARN best practices.

Problem Context

After migrating from Hadoop v1 to YARN, a user encountered container errors when running the same MapReduce application on identical hardware (8GB RAM, 8 processors). In Hadoop v1, the user allocated 1GB memory per mapper and reducer slot, with tasks running smoothly. However, in YARN, containers were terminated for exceeding memory limits despite basic configuration.

Initial settings (yarn-site.xml):

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
</property>
<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>8192</value>
</property>
<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>8192</value>
</property>

The error indicated container exceeded virtual memory limits:

Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

The user then increased memory allocation in mapred-site.xml:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>4096</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>4096</value>
</property>

Yet, the issue persisted with containers killed for exceeding physical memory:

Container [pid=26783,containerID=container_1389136889967_0009_01_000002] is running beyond physical memory limits. Current usage: 4.2 GB of 4 GB physical memory used; 5.2 GB of 8.4 GB virtual memory used. Killing container.

Core Concept Analysis

YARN, as the resource manager in Hadoop v2, introduces finer-grained memory control. Key configuration parameters include:

• yarn.scheduler.minimum-allocation-mb: Minimum memory a container can request.
• yarn.scheduler.maximum-allocation-mb: Maximum memory a container can request.
• yarn.nodemanager.resource.memory-mb: Total memory available for containers on a node manager.
• mapreduce.map.memory.mb and mapreduce.reduce.memory.mb: Memory requested by Map and Reduce tasks.

However, these parameters only define upper limits for memory requests; actual usage is also influenced by JVM heap size and virtual memory checks.

Solution

Based on the best answer, a complete configuration in mapred-site.xml should include:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>4096</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>8192</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx3072m</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx6144m</value>
</property>

Here, mapreduce.map.java.opts and mapreduce.reduce.java.opts set JVM heap size limits, ensuring they are below container-allocated memory to avoid physical memory breaches. For example, a Map task container is allocated 4GB, but the JVM heap is capped at 3GB, reserving space for the OS and other processes.

Virtual Memory Handling

Supplementary answers note that virtual memory issues are prominent on systems like CentOS/RHEL 6 due to aggressive allocation strategies. Solutions include:

1. Disable virtual memory checks: Set yarn.nodemanager.vmem-check-enabled to false in yarn-site.xml.
2. Adjust virtual-to-physical memory ratio: Set yarn.nodemanager.vmem-pmem-ratio to a higher value (e.g., 4).

Example configuration:

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
  <description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
  <name>yarn.nodemanager.vmem-pmem-ratio</name>
  <value>4</value>
  <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

Performance Optimization Recommendations

The user observed that increasing container memory led to higher usage, possibly related to data splitting strategies. In YARN, container size affects task parallelism. Smaller containers can increase parallel tasks but must have sufficient resources for their data blocks. Recommendations:

• Monitor task memory usage patterns to adjust mapreduce.map.memory.mb and mapreduce.reduce.memory.mb to actual needs.
• Use YARN resource scheduler logs to analyze container resource utilization.
• Consider data locality to avoid excessive data movement with overly small containers.

Conclusion

Migrating to YARN requires shifting from simple slot allocation to precise container management. Key steps include correctly setting MapReduce memory requests, configuring JVM heap size limits, and handling virtual memory checks. With this article's approach, users can resolve container memory limit issues, optimize resource utilization, and ensure stable MapReduce application performance on YARN. Practice shows that combining physical and virtual memory configurations effectively balances performance and resource constraints.