Apache Flink 中文用户邮件列表

flink on k8s native 测试失败

Classic

List

Threaded

5 messages Options

仙剑……情动人间

flink on k8s native 测试失败

Hi All，

我这里在自己搭建的 k8s 测试环境中按照官网的 flink on k8s native 指南进行测试，然后一直报错，具体情况如下，请求大佬们的帮助，不胜感激。

一、环境

k8s v1.18.16

flink flink-1.12.2

java openjdk version "11" 2018-09-25

二、官网规定的前提条件

1、Kubernetes >= 1.9 满足

2、权限检查满足

3、启用 Kubernetes DNS 满足

4、default service account with RBAC permissions to create, delete pods ，我这里创建新的命名空间和账号，也是满足的，情况如下：

kubectl create namespace flink-session-cluster

kubectl create serviceaccount flink -n flink-session-cluster

kubectl create clusterrolebinding flink-role-binding-flink \ --clusterrole=edit \ --serviceaccount=flink-session-cluster:flink

三、启动 session 并提交 flink 任务

3.1 启动session 命令

./bin/kubernetes-session.sh -Dkubernetes.cluster-id=my-first-flink-cluster \

> -Dkubernetes.namespace=flink-session-cluster \

> -Dkubernetes.jobmanager.service-account=flink \

> -Dkubernetes.service.exposed.type=NodePort

日志：

3.2 提交flink 任务

命令：

./bin/flink run \

> --target kubernetes-session \

> -Dkubernetes.namespace=flink-session-cluster \

> -Dkubernetes.cluster-id=my-first-flink-cluster \

> -Dkubernetes.jobmanager.service-account=flink \

> ./examples/streaming/TopSpeedWindowing.jar

报错日志如下：

Executing TopSpeedWindowing example with default input data set.

Use --input to specify file input.

Printing result to stdout. Use --output to specify output path.

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/home/dsi/soft/flink/flink-1.12.2/lib/flink-dist_2.11-1.12.2.jar) to field java.lang.String.value

WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

2021-05-12 15:51:30,453 INFO org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Retrieve flink cluster my-first-flink-cluster successfully, JobManager Web Interface: http://10.199.252.101:8081

------------------------------------------------------------

The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'CarTopSpeedWindowingExample'.

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)

at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)

at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)

at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)

at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)

at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)

at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)

Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'CarTopSpeedWindowingExample'.

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1918)

at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)

at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)

at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1782)

at org.apache.flink.streaming.examples.windowing.TopSpeedWindowing.main(TopSpeedWindowing.java:99)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)

... 8 more

Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.

at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:400)

at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)

at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)

at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)

at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)

at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:390)

at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)

at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)

at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)

at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)

at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:430)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.

at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:386)

... 21 more

Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.199.252.101:8081

at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)

at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)

at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)

... 19 more

Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.199.252.101:8081

Caused by: java.net.ConnectException: Connection refused

at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)

at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)

at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)

at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at java.base/java.lang.Thread.run(Thread.java:834)

Best，

Luke Yan

Yang Wang

Re: flink on k8s native 测试失败

你参数配置错了，应该是kubernetes.rest-service.exposed.type=NodePort

Best,
Yang

仙剑……情动人间 <[hidden email]> 于2021年5月12日周三上午9:45写道：

> Hi All，
>
> 我这里在自己搭建的 k8s 测试环境中按照官网的 flink on k8s native
> 指南进行测试，然后一直报错，具体情况如下，请求大佬们的帮助，不胜感激。
>
> 一、环境
>
> k8s v1.18.16
> flink flink-1.12.2
> java openjdk version "11" 2018-09-25
>
> 二、官网规定的前提条件
>
> 1、Kubernetes >= 1.9 满足
>
>
>
> 2、权限检查满足
>
>
> 3、启用 Kubernetes DNS 满足
>
>
> 4、default service account with RBAC
> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#rbac> permissions
> to create, delete pods ，我这里创建新的命名空间和账号，也是满足的，情况如下：
>
> kubectl create namespace flink-session-cluster
>
> kubectl create serviceaccount flink -n flink-session-cluster
>
> kubectl create clusterrolebinding flink-role-binding-flink \ --clusterrole
> =edit \ --serviceaccount=flink-session-cluster:flink
>
>
> 三、启动 session 并提交 flink 任务
>
> 3.1 启动session 命令
>
> ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=my-first-flink-cluster
> \
> > -Dkubernetes.namespace=flink-session-cluster \
> > -Dkubernetes.jobmanager.service-account=flink \
> > -Dkubernetes.service.exposed.type=NodePort
>
> 日志：
>
>
> 3.2 提交flink 任务
>
> 命令：
> ./bin/flink run \
> > --target kubernetes-session \
> > -Dkubernetes.namespace=flink-session-cluster \
> > -Dkubernetes.cluster-id=my-first-flink-cluster \
> > -Dkubernetes.jobmanager.service-account=flink \
> > ./examples/streaming/TopSpeedWindowing.jar
>
> 报错日志如下：
>
> Executing TopSpeedWindowing example with default input data set.
> Use --input to specify file input.
> Printing result to stdout. Use --output to specify output path.
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.flink.api.java.ClosureCleaner
> (file:/home/dsi/soft/flink/flink-1.12.2/lib/flink-dist_2.11-1.12.2.jar) to
> field java.lang.String.value
> WARNING: Please consider reporting this to the maintainers of
> org.apache.flink.api.java.ClosureCleaner
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> 2021-05-12 15:51:30,453 INFO
> org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Retrieve
> flink cluster my-first-flink-cluster successfully, JobManager Web
> Interface: http://10.199.252.101:8081
>
> ------------------------------------------------------------
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error: Failed to execute job 'CarTopSpeedWindowingExample'.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: org.apache.flink.util.FlinkException: Failed to execute job
> 'CarTopSpeedWindowingExample'.
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1918)
> at
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)
> at
> org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1782)
> at
> org.apache.flink.streaming.examples.windowing.TopSpeedWindowing.main(TopSpeedWindowing.java:99)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
> ... 8 more
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed
> to submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:400)
> at
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> at
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:390)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:430)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:386)
> ... 21 more
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: /10.199.252.101:8081
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
> ... 19 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: /10.199.252.101:8081
> Caused by: java.net.ConnectException: Connection refused
> at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.base/java.lang.Thread.run(Thread.java:834)
>
>
> Best，
>
> Luke Yan
>
>

仙剑……情动人间

回复： flink on k8s native 测试失败

Hi Yang，

感谢您的指导，我已经成功完成flink session on k8s 的examples 测试，但是在使用 application 模式的时候仍然失败，详情如下：

环境如之前所述：

提交任务的命令：

flink run-application \

--target kubernetes-application \

-Dkubernetes.namespace=flink-session-cluster \

-Dkubernetes.cluster-id=my-first-application-cluster \

-Dkubernetes.container.image=flink:latest \

local:///opt/flink/examples/batch/WordCount.jar

查看 deployment 状态如图所示：

查看service状态如图所示

查看pod状态：

然后 describe pods 信息如下：

如上图，提示的是找不到 configmap flink-config-my-first-application-cluster导致挂载失败，但是我使用 kubectl describe 又能够看到 flink-config-my-first-application-cluster，如下图所示

如下为 describe pods 详细信息：

Name: my-first-application-cluster-6c85b474-64t6x

Namespace: flink-session-cluster

Priority: 0

Node: hkctttl104/10.199.252.104

Start Time: Thu, 13 May 2021 22:16:03 +0800

Labels: app=my-first-application-cluster

component=jobmanager

pod-template-hash=6c85b474

type=flink-native-kubernetes

Annotations: <none>

Status: Running

IP: 10.47.0.8

IPs:

IP: 10.47.0.8

Controlled By: ReplicaSet/my-first-application-cluster-6c85b474

Containers:

flink-job-manager:

Container ID: docker://0325efde4632560749515c225a89775414fb8e96dbd10757ede2b9e381c551f8

Image: flink:latest

Image ID: docker-pullable://flink@sha256:4911cd2b70f87a5286803fffb547c3676b7c3d078206547a4354be6749b1eef2

Ports: 8081/TCP, 6123/TCP, 6124/TCP

Host Ports: 0/TCP, 0/TCP, 0/TCP

Command:

/docker-entrypoint.sh

Args:

native-k8s

$JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 -Dlog.file=/opt/flink/log/jobmanager.log -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=201326592b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1073741824b -D jobmanager.memory.jvm-overhead.max=201326592b

State: Waiting

Reason: CrashLoopBackOff

Last State: Terminated

Reason: Error

Exit Code: 1

Started: Thu, 13 May 2021 22:27:59 +0800

Finished: Thu, 13 May 2021 22:27:59 +0800

Ready: False

Restart Count: 6

Limits:

cpu: 1

memory: 1600Mi

Requests:

cpu: 1

memory: 1600Mi

Environment:

_POD_IP_ADDRESS: (v1:status.podIP)

Mounts:

/opt/flink/conf from flink-config-volume (rw)

/var/run/secrets/kubernetes.io/serviceaccount from default-token-9qxvx (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

flink-config-volume:

Type: ConfigMap (a volume populated by a ConfigMap)

Name: flink-config-my-first-application-cluster

Optional: false

default-token-9qxvx:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-9qxvx

Optional: false

QoS Class: Guaranteed

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s

node.kubernetes.io/unreachable:NoExecute for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 14m default-scheduler Successfully assigned flink-session-cluster/my-first-application-cluster-6c85b474-64t6x to hkctttl104

Warning FailedMount 14m kubelet MountVolume.SetUp failed for volume "flink-config-volume" : configmap "flink-config-my-first-application-cluster" not found

Normal Pulling 14m kubelet Pulling image "flink:latest"

Normal Pulled 8m46s kubelet Successfully pulled image "flink:latest"

Normal Created 7m8s (x5 over 8m46s) kubelet Created container flink-job-manager

Normal Started 7m8s (x5 over 8m45s) kubelet Started container flink-job-manager

Normal Pulled 7m8s (x4 over 8m45s) kubelet Container image "flink:latest" already present on machine

Warning BackOff 4m36s (x23 over 8m43s) kubelet Back-off restarting failed container

求助各位大佬给点提示，不剩感激

Best，

Luke Yan

------------------ 原始邮件 ------------------

发件人: "user-zh" <[hidden email]>;

发送时间: 2021年5月12日(星期三) 下午5:42

收件人: "user-zh"<[hidden email]>;

主题: Re: flink on k8s native 测试失败

你参数配置错了，应该是kubernetes.rest-service.exposed.type=NodePort

Best,
Yang

仙剑……情动人间 <[hidden email]> 于2021年5月12日周三上午9:45写道：

> Hi All，
>
> 我这里在自己搭建的 k8s 测试环境中按照官网的 flink on k8s native
> 指南进行测试，然后一直报错，具体情况如下，请求大佬们的帮助，不胜感激。
>
> 一、环境
>
>     k8s v1.18.16
>     flink flink-1.12.2
>     java openjdk version "11" 2018-09-25
>
> 二、官网规定的前提条件
>
>    1、Kubernetes >= 1.9 满足
>
>
>
>     2、权限检查满足
>
>
> 3、启用 Kubernetes DNS 满足
>
>
> 4、default service account with RBAC
> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#rbac> permissions
> to create, delete pods ，我这里创建新的命名空间和账号，也是满足的，情况如下：
>
> kubectl create namespace flink-session-cluster
>
> kubectl create serviceaccount flink -n flink-session-cluster
>
> kubectl create clusterrolebinding flink-role-binding-flink \ --clusterrole
> =edit \ --serviceaccount=flink-session-cluster:flink
>
>
> 三、启动 session 并提交 flink 任务
>
>      3.1 启动session 命令
>
> ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=my-first-flink-cluster
> \
> > -Dkubernetes.namespace=flink-session-cluster \
> > -Dkubernetes.jobmanager.service-account=flink \
> > -Dkubernetes.service.exposed.type=NodePort
>
>   日志：
>
>
>   3.2 提交flink 任务
>
>    命令：
>       ./bin/flink run \
> >     --target kubernetes-session \
> > -Dkubernetes.namespace=flink-session-cluster \
> >     -Dkubernetes.cluster-id=my-first-flink-cluster \
> > -Dkubernetes.jobmanager.service-account=flink \
> >     ./examples/streaming/TopSpeedWindowing.jar
>
>    报错日志如下：
>
> Executing TopSpeedWindowing example with default input data set.
> Use --input to specify file input.
> Printing result to stdout. Use --output to specify output path.
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.flink.api.java.ClosureCleaner
> (file:/home/dsi/soft/flink/flink-1.12.2/lib/flink-dist_2.11-1.12.2.jar) to
> field java.lang.String.value
> WARNING: Please consider reporting this to the maintainers of
> org.apache.flink.api.java.ClosureCleaner
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> 2021-05-12 15:51:30,453 INFO
> org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Retrieve
> flink cluster my-first-flink-cluster successfully, JobManager Web
> Interface: http://10.199.252.101:8081
>
> ------------------------------------------------------------
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error: Failed to execute job 'CarTopSpeedWindowingExample'.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: org.apache.flink.util.FlinkException: Failed to execute job
> 'CarTopSpeedWindowingExample'.
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1918)
> at
> org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)
> at
> org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1782)
> at
> org.apache.flink.streaming.examples.windowing.TopSpeedWindowing.main(TopSpeedWindowing.java:99)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
> ... 8 more
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed
> to submit JobGraph.
> at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:400)
> at
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> at
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:390)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:430)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:386)
> ... 21 more
> Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: /10.199.252.101:8081
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
> ... 19 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: /10.199.252.101:8081
> Caused by: java.net.ConnectException: Connection refused
> at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.base/java.lang.Thread.run(Thread.java:834)
>
>
> Best，
>
> Luke Yan
>
>

Lori681

Re: flink on k8s native 测试失败

In reply to this post by 仙剑……情动人间

Lori681

Re: flink on k8s native 测试失败

In reply to this post by 仙剑……情动人间