任务配置 (Hadoop 2.2.0)

任务 JVM 内存设置 (AMI 3.0.0) hadoop 2.2.0 使用两个参数配置用于 map 和 reduce 的内存：分别为 mapreduce.map.java.opts 和 mapreduce.reduce.java.opts。它们取代了早期 hadoop 版本中的单个配置选项：mapreduce.map.java.opts。 下表显示了每实例类型 任务 JVM 内存设置 (AMI 3.0.0) hadoop 2.2.0 使用两个参数配置用于 map 和 reduce 的内存：分别为 mapreduce.map.java.opts 和 mapreduce.reduce.java.opts。它们取代了早期 hadoop 版本中的单个配置选项：mapreduce.map.java.opts。 下表显示了每实例类型的这些设置的默认值。 m1.medium 配置选项默认值mapreduce.map.java.opts-Xmx768mmapreduce.reduce.java.opts-Xmx768mmapreduce.map.memory.mb1024mapreduce.reduce.memory.mb1024yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb2048yarn.nodemanager.resource.memory-mb2048 m1.large 配置选项默认值mapreduce.map.java.opts-Xmx864mmapreduce.reduce.java.opts-Xmx1536mmapreduce.map.memory.mb1024mapreduce.reduce.memory.mb2048yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb3072yarn.nodemanager.resource.memory-mb5120 m1.xlarge 配置选项默认值mapreduce.map.java.opts-Xmx512mmapreduce.reduce.java.opts-Xmx1536mmapreduce.map.memory.mb768mapreduce.reduce.memory.mb2048yarn.scheduler.minimum-allocation-mb256yarn.scheduler.maximum-allocation-mb8 192yarn.nodemanager.resource.memory-mb12288 m2.xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1280mmapreduce.reduce.java.opts-Xmx2304mmapreduce.map.memory.mb1536mapreduce.reduce.memory.mb2560yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb7168yarn.nodemanager.resource.memory-mb14336 m2.2xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1280mmapreduce.reduce.java.opts-Xmx2304mmapreduce.map.memory.mb1536mapreduce.reduce.memory.mb2560yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb8 192yarn.nodemanager.resource.memory-mb30720 m3.xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1515mmapreduce.reduce.java.opts-Xmx1792mmapreduce.map.memory.mb1904mapreduce.reduce.memory.mb2150yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb3788yarn.nodemanager.resource.memory-mb5273 m3.2xlarge 配置选项默认值mapreduce.map.java.opts-Xmx2129mmapreduce.reduce.java.opts-Xmx2560mmapreduce.map.memory.mb2826mapreduce.reduce.memory.mb3072yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb5324yarn.nodemanager.resource.memory-mb9113 m2.4xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1280mmapreduce.reduce.java.opts-Xmx2304mmapreduce.map.memory.mb1536mapreduce.reduce.memory.mb2560yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb8 192yarn.nodemanager.resource.memory-mb61440 c1.medium 配置选项默认值io.sort.mb100mapreduce.map.java.opts-Xmx288mmapreduce.reduce.java.opts-Xmx288mmapreduce.map.memory.mb512mapreduce.reduce.memory.mb512yarn.scheduler.minimum-allocation-mb256yarn.scheduler.maximum-allocation-mb512yarn.nodemanager.resource.memory-mb1024 c1.xlarge 配置选项默认值io.sort.mb150mapreduce.map.java.opts-Xmx864mmapreduce.reduce.java.opts-Xmx1536mmapreduce.map.memory.mb1024mapreduce.reduce.memory.mb2048yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb2048yarn.nodemanager.resource.memory-mb5120 c3.large 配置选项默认值mapreduce.map.java.opts-Xmx768mmapreduce.reduce.java.opts-Xmx768mmapreduce.map.memory.mb921mapreduce.reduce.memory.mb921yarn.scheduler.minimum-allocation-mb499yarn.scheduler.maximum-allocation-mb1920yarn.nodemanager.resource.memory-mb1920 c3.xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1177mmapreduce.reduce.java.opts-Xmx1356mmapreduce.map.memory.mb1413mapreduce.reduce.memory.mb1628yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb2944yarn.nodemanager.resource.memory-mb3302 c3.2xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1515mmapreduce.reduce.java.opts-Xmx1792mmapreduce.map.memory.mb1904mapreduce.reduce.memory.mb2150yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb3788yarn.nodemanager.resource.memory-mb5273 c3.4xlarge 配置选项默认值mapreduce.map.java.opts-Xmx2129mmapreduce.reduce.java.opts-Xmx2560mmapreduce.map.memory.mb2826mapreduce.reduce.memory.mb3072yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb5324yarn.nodemanager.resource.memory-mb9113 c3.8xlarge 配置选项默认值mapreduce.map.java.opts-Xmx4669mmapreduce.reduce.java.opts-Xmx4915mmapreduce.map.memory.mb4669mapreduce.reduce.memory.mb4915yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb8396yarn.nodemanager.resource.memory-mb16793 cc1.4xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1280mmapreduce.reduce.java.opts-Xmx2304mmapreduce.map.memory.mb1536mapreduce.reduce.memory.mb2560yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb5120yarn.nodemanager.resource.memory-mb20480 cg1.4xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1280mmapreduce.reduce.java.opts-Xmx2304mmapreduce.map.memory.mb1536mapreduce.reduce.memory.mb2560yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb5120yarn.nodemanager.resource.memory-mb20480 cc2.8xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1280mmapreduce.reduce.java.opts-Xmx2304mmapreduce.map.memory.mb1536mapreduce.reduce.memory.mb2560yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb8 192yarn.nodemanager.resource.memory-mb56320 cr1.8xlarge 配置选项默认值mapreduce.map.java.opts-Xmx10895mmapreduce.reduce.java.opts-Xmx13516mmapreduce.map.memory.mb15974mapreduce.reduce.memory.mb16220yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb27238yarn.nodemanager.resource.memory-mb63897 g2.2xlarge 配置选项默认值mapreduce.map.java.opts-Xmx512mmapreduce.reduce.java.opts-Xmx1536mmapreduce.map.memory.mb768mapreduce.reduce.memory.mb2048yarn.scheduler.minimum-allocation-mb256yarn.scheduler.maximum-allocation-mb8 192yarn.nodemanager.resource.memory-mb12288 hi1.4xlarge 配置选项默认值mapreduce.map.java.opts-Xmx3379mmapreduce.reduce.java.opts-Xmx4121mmapreduce.map.memory.mb4700mapreduce.reduce.memory.mb4945yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb8448yarn.nodemanager.resource.memory-mb16921 hs1.8xlarge 配置选项默认值mapreduce.map.java.opts-Xmx1280mmapreduce.reduce.java.opts-Xmx2304mmapreduce.map.memory.mb1536mapreduce.reduce.memory.mb2560yarn.scheduler.minimum-allocation-mb512yarn.scheduler.maximum-allocation-mb8 192yarn.nodemanager.resource.memory-mb56320 i2.xlarge 配置选项默认值mapreduce.map.java.opts-Xmx2150mmapreduce.reduce.java.opts-Xmx2585mmapreduce.map.memory.mb2856mapreduce.reduce.memory.mb3102yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb5376yarn.nodemanager.resource.memory-mb9241 i2.2xlarge 配置选项默认值mapreduce.map.java.opts-Xmx3399mmapreduce.reduce.java.opts-Xmx4147mmapreduce.map.memory.mb4730mapreduce.reduce.memory.mb4976yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb8499yarn.nodemanager.resource.memory-mb17049 i2.4xlarge 配置选项默认值mapreduce.map.java.opts-Xmx5898mmapreduce.reduce.java.opts-Xmx7270mmapreduce.map.memory.mb8478mapreduce.reduce.memory.mb8724yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb14745yarn.nodemanager.resource.memory-mb32665 i2.8xlarge 配置选项默认值mapreduce.map.java.opts-Xmx10895mmapreduce.reduce.java.opts-Xmx13516mmapreduce.map.memory.mb15974mapreduce.reduce.memory.mb16220yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb27238yarn.nodemanager.resource.memory-mb63897 r3.xlarge 配置选项默认值mapreduce.map.java.opts-Xmx2150mmapreduce.reduce.java.opts-Xmx3102mmapreduce.map.memory.mb2856mapreduce.reduce.memory.mb3102yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb5376yarn.nodemanager.resource.memory-mb9241 r3.2xlarge 配置选项默认值mapreduce.map.java.opts-Xmx3399mmapreduce.reduce.java.opts-Xmx4147mmapreduce.map.memory.mb4743mapreduce.reduce.memory.mb4976yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb8499yarn.nodemanager.resource.memory-mb17049 r3.4xlarge 配置选项默认值mapreduce.map.java.opts-Xmx5898mmapreduce.reduce.java.opts-Xmx7270mmapreduce.map.memory.mb8478mapreduce.reduce.memory.mb8724yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb14745yarn.nodemanager.resource.memory-mb32665 r3.8xlarge 配置选项默认值mapreduce.map.java.opts-Xmx10895mmapreduce.reduce.java.opts-Xmx13516mmapreduce.map.memory.mb15974mapreduce.reduce.memory.mb16220yarn.scheduler.minimum-allocation-mb532yarn.scheduler.maximum-allocation-mb27238yarn.nodemanager.resource.memory-mb69897 您可以在执行每项任务时启动新 JVM，实现更好的任务隔离；也可以在各项任务之间共享 JVM，降低框架开销。如果您处理的是许多小文件，合理的做法是多次重复使用 JVM，以摊销启动成本。然而，如果每项任务耗时较长或处理的数据量较大，您可以选择不重复使用 JVM，以确保释放出所有内存供后续任务使用。 使用 选项配置 JVM 重复使用设置。 使用引导操作修改 JVM 在安装了 Amazon EMR CLI 的目录下，从命令行运行以下命令。有关更多信息，请参见 Amazon EMR 的命令行接口参考。 Linux、UNIX 和 Mac OS X 用户： Windows 用户： Note Amazon EMR 已将 的值设置为 20，但您可以通过引导操作覆盖此值。值 表示在单一作业中重复使用的次数不受限制， 表示不能重复使用任务。