问题描述
配置好Hadoop集群(包括hdfs和yarn)，配置好spark-on-yarn，提交任务后发现container异常退出，有core dump产生；修改yarn的资源配置，依然crash。
container日志一旦任务结束就被系统自动清除，只能通过当前控制台分析日志；日志中有core dump字样和java error report的具体地址 /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1523510046494_0001/container_1523510046494_0001_01_000002/hs_err_pid4883.log，但去查看却看不到这个文件。
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc: SuppressErrorAt=/memnode.cpp:2307#
# A fatal error has been detected by the Java Runtime Environment:#
# Internal Error (/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/openjdk/hotspot/src/share/vm/opto/memnode.cpp:2307), pid=4883, tid=0x00007f930d7f6700
# assert(Opcode() == mem->Opcode() || phase->C->get_alias_index(adr_type()) == Compile::AliasIdxRaw) failed: no mismatched stores, except on raw memory
#
# JRE version: OpenJDK Runtime Environment (8.0_161-b14) (build 1.8.0_161-debug-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.161-b14-debug mixed mode linux-amd64 compressed oops)
#Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
#/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1523510046494_0001/container_1523510046494_0001_01_000002/hs_err_pid4883.log
#
# Compiler replay data is saved as:
# /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1523510046494_0001/container_1523510046494_0001_01_000002/replay_pid4883.log
#
# If you would like to submit a bug report, please visit:
#http://bugreport.java.com/bugreport/crash.jsp#
Current thread is 140269563373312
Dumping core ...
解决方式
首先修改‘yarn.nodemanager.delete.debug-delay-sec’配置，保留日志文件
yarn-site.xml yarn.nodemanager.delete.debug-delay-sec 360000
Error report里看到了错误信息， 执行的JVM的源代码
nosuchmethoderror openjdk/hotspot/src/share/vm/prims/jni.cpp
开始怀疑是JDK的版本，安装的是debug版本。java-1.8.0-openjdk有如下版本可以安装，重新安装为java-1.8.0-openjdk-devel.x86_64，问题解决。程序能够稳定执行了，速度也变快了。
java-1.8.0-openjdk.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-devel.x86_64 : OpenJDK Development Environment
java-1.8.0-openjdk-debug.x86_64 : OpenJDK Runtime Environment with full debug on
java-1.8.0-openjdk-devel-debug.x86_64 : OpenJDK Development Environment with full debug on
