环境描述
- spark 版本: spark-2.4.0-bin-hadoop2.7
- hdp 版本: 2.6.3.0-235
Spark整合Hadoop
- cp spark-env.sh.template spark-env.sh
- vim spark-env.sh
HADOOP_CONF_DIR=/usr/hdp/current/hadoop-client/etc/hadoop
YARN_CONF_DIR=/usr/hdp/current/hadoop-client/etc/hadoop
提交任务时的异常解决
异常
- Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.server.spi.Container
- Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
- YARN LOG: java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()Ljava/util/Map;
- 命令行错误: Container exited with a non-zero exit code 1 && YARN LOG: ... bad substitution
解决问题1
- 原因: 复制jersey依赖,统一spark和hadoop的jersey-client和jersey-core版本
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/jars
cp /usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar .
cp /usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar .
解决问题2
- 原因: hadoop依赖不统一, 替换hadoop-yarn相关依赖
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/jars
rm -f ./hadoop-yarn*
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.6.3.0-235.jar .
cp /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.6.3.0-235.jar .
解决问题3
- 原因: jsr305依赖版本不统一, 替换jsr305依赖
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/jars
rm -f ./jsr305-1.3.9.jar
cp /usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar .
解决问题4
- 原因: hdp版本变量未设置
- 查看hdp版本
hdp-select status hadoop-client
image.png
-
添加变量
cd /home/wcm/apps/spark-2.4.0-bin-hadoop2.7/conf cp spark-defaults.conf.template spark-defaults.conf-
vim spark-defaults.conf
spark.driver.extraJavaOptions -Dhdp.version=2.6.3.0-235 spark.executor.extraJavaOptions -Dhdp.version=2.6.3.0-235 spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.3.0-235 -
vim java-opts
-Dhdp.version=2.6.3.0-235
-
