Re: Flink 1.10.0 on yarn 提交job失败

[Xintong Song]

看起来是 Yarn 没有给应用设置 hadoop classpath。可以登机器确认一下 launch_container.sh
的内容，container 启动命令里是否包含了正确的 hadoop classpath。Yarn 是定制过的版本吗？按理说开源版本都会给
container 设置 hadoop classpath 的。

1.10 版本以前可以运行是因为 flink 自带了 shaded hadoop，从 1.10 版本开始 flink 默认不再携带 shaded
hadoop，而是使用集群环境的 hadoop 依赖。你也可以自己携带 shaded hadoop[1]，应该也可以运行。

Thank you~

Xintong Song


[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/hadoop.html#adding-hadoop-to-lib

On Sat, Dec 12, 2020 at 8:05 PM Jacob <[hidden email]> wrote:

> Hello, 请问在flink 1.10.0 on yarn提交job出现此问题是什么原因，hadoop
> jar包依赖吗？该程序在1.10以下的版本均可运行，在1.10.0无法提交。
>
> 谢谢！
> ================================================================
>
> [jacob@hadoop001 bin]$ ./yarn logs -applicationId
> application_1603495749855_57650
> 20/12/11 18:52:55 INFO client.RMProxy: Connecting to ResourceManager at
> localhost:8032
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/opt/app/hadoop_client/e11_backend/hadoop-2.6.0-cdh5.8.3/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/opt/app/hadoop-2.6.0-cdh5.8.3/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 20/12/11 18:52:57 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
>
> Container: container_1603495749855_57650_02_000001 on localhost
>
> =====================================================================================
> LogType:jobmanager.err
> Log Upload Time:Fri Dec 11 18:49:21 -0800 2020
> LogLength:2368
> Log Contents:
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>
> [jar:file:/data/hadoop/dn/sdc/yarn/nm/usercache/jacob/appcache/application_1603495749855_57650/filecache/11/datafeed-website-filter_flink-0.0.1-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/data/hadoop/dn/sde/yarn/nm/usercache/jacob/appcache/application_1603495749855_57650/filecache/17/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>
> [jar:file:/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
> Exception in thread "main" java.lang.NoSuchMethodError:
>
> org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
>         at
>
> org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:54)
>         at
>
> org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:42)
>         at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:119)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:348)
>         at
>
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1659)
>         at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:91)
>         at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
>         at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>         at org.apache.hadoop.security.Groups.<init>(Groups.java:55)
>         at
>
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
>         at
>
> org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:136)
>         at
>
> org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:109)
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/
>