Apache Flink 中文用户邮件列表

Re: flink 1.11.2 on yarn 可用slot始终为0，job无法提交

Classic

List

Threaded

3 messages Options

Xintong Song

Re: flink 1.11.2 on yarn 可用slot始终为0，job无法提交

jobmanager 的日志方便发下吗？
另外，可以看下 yarn 是否分配了 taskmanager 的 container，如果有的话通过 yarn 获取以下 taskmanager
的日志。

Thank you~

Xintong Song

On Thu, Dec 10, 2020 at 9:55 AM Jacob <[hidden email]> wrote:

> <
> http://apache-flink.147419.n8.nabble.com/file/t1162/Screenshot_2020-12-09_153858.png>
>
>
>
> 启动命令：
> ./bin/flink run-application -t yarn-application
> -Djobmanager.memory.process.size=2048m
> -Dtaskmanager.memory.process.size=2048m -Dyarn.application.name="Test Job"
> -c com.jacob.Main /opt/app/test.jar
>
> Hadoop集群资源充足。flink无法为job分配slot。
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/
>

Jacob

Re: flink 1.11.2 on yarn 可用slot始终为0，job无法提交

*Thank you for your reply！*

日志以及pom文件如下
Container: container_1603495749855_55197_02_000001 on hadoop01
=====================================================================================
LogType:jobmanager.err
Log Upload Time:Wed Dec 09 17:03:38 -0800 2020
LogLength:802
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/data/hadoop/dn/sde/yarn/nm/usercache/jacob/appcache/application_1603495749855_55197/filecache/26/test_job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/data/hadoop/dn/sdd/yarn/nm/usercache/jacob/appcache/application_1603495749855_55197/filecache/33/test_job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]

LogType:jobmanager.log
Log Upload Time:Wed Dec 09 17:03:38 -0800 2020
LogLength:980
Log Contents:
2020-12-09 17:03:31,918 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19140-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2020-12-09 17:03:31,931 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19140-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.keystores.factory.class;
Ignoring.
2020-12-09 17:03:31,931 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19140-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
2020-12-09 17:03:31,932 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19140-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.

LogType:jobmanager.out
Log Upload Time:Wed Dec 09 17:03:38 -0800 2020
LogLength:2188
Log Contents:
2020-12-09 17:03:36.375 [main] ERROR o.a.f.r.entrypoint.ClusterEntrypoint -
Could not start cluster entrypoint YarnJobClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to
initialize the cluster entrypoint YarnJobClusterEntrypoint.
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:520)
at
org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
at
org.apache.hadoop.yarn.conf.YarnConfiguration.addDeprecatedKeys(YarnConfiguration.java:79)
at
org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:73)
at
org.apache.flink.yarn.YarnResourceManager.<init>(YarnResourceManager.java:155)
at
org.apache.flink.yarn.entrypoint.YarnResourceManagerFactory.createResourceManager(YarnResourceManagerFactory.java:76)
at
org.apache.flink.runtime.resourcemanager.ResourceManagerFactory.createResourceManager(ResourceManagerFactory.java:61)
at
org.apache.flink.runtime.resourcemanager.ActiveResourceManagerFactory.createResourceManager(ActiveResourceManagerFactory.java:58)
at
org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:167)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:219)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171)
... 2 common frames omitted

Container: container_1603495749855_55197_01_000001 on hadoop01
=====================================================================================
LogType:jobmanager.err
Log Upload Time:Wed Dec 09 17:03:37 -0800 2020
LogLength:802
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/data/hadoop/dn/sdc/yarn/nm/usercache/jacob/appcache/application_1603495749855_55197/filecache/26/test_job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/data/hadoop/dn/sdf/yarn/nm/usercache/jacob/appcache/application_1603495749855_55197/filecache/33/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]

LogType:jobmanager.log
Log Upload Time:Wed Dec 09 17:03:37 -0800 2020
LogLength:980
Log Contents:
2020-12-09 17:03:16,286 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19167-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2020-12-09 17:03:16,297 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19167-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.keystores.factory.class;
Ignoring.
2020-12-09 17:03:16,297 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19167-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
2020-12-09 17:03:16,298 WARN org.apache.hadoop.conf.Configuration
[] - /run/cloudera-scm-agent/process/19167-yarn-NODEMANAGER/core-site.xml:an
attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.

LogType:jobmanager.out
Log Upload Time:Wed Dec 09 17:03:37 -0800 2020
LogLength:2188
Log Contents:
2020-12-09 17:03:20.766 [main] ERROR o.a.f.r.entrypoint.ClusterEntrypoint -
Could not start cluster entrypoint YarnJobClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to
initialize the cluster entrypoint YarnJobClusterEntrypoint.
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:520)
at
org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
at
org.apache.hadoop.yarn.conf.YarnConfiguration.addDeprecatedKeys(YarnConfiguration.java:79)
at
org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:73)
at
org.apache.flink.yarn.YarnResourceManager.<init>(YarnResourceManager.java:155)
at
org.apache.flink.yarn.entrypoint.YarnResourceManagerFactory.createResourceManager(YarnResourceManagerFactory.java:76)
at
org.apache.flink.runtime.resourcemanager.ResourceManagerFactory.createResourceManager(ResourceManagerFactory.java:61)
at
org.apache.flink.runtime.resourcemanager.ActiveResourceManagerFactory.createResourceManager(ActiveResourceManagerFactory.java:58)
at
org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:167)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:219)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171)
... 2 common frames omitted

*【pom文件】*

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>***</groupId>
<artifactId>***</artifactId>
<version>0.0.1-SNAPSHOT</version>

<name>***</name>
<url>***</url>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<flink.version>1.11.2</flink.version>
<scala.binary.version>2.11</scala.binary.version>
<kafka.version>0.10</kafka.version>
<hive.version>1.1.0-cdh5.8.3</hive.version>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
<version>1.5.4</version>
</dependency>
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson</artifactId>
<version>3.12.4</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>6.2.2.jre8</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>javax.mail</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>

<artifactId>flink-connector-kafka-${kafka.version}_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>

<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>

<artifactId>flink-connector-filesystem_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.30</version>
</dependency>
</dependencies>
</dependencyManagement>

<build>
********************
</build>
</project>

Thanks！
Jacob

--
Sent from: http://apache-flink.147419.n8.nabble.com/

Thanks!
Jacob

Lori681

Re: flink 1.11.2 on yarn 可用slot始终为0，job无法提交

In reply to this post by Xintong Song