大佬们,请教个问题,我是在k8s上部署flink1.11 natvie session模式。
下载的flink-1.11.0-bin-scala_2.11.tgz 执行命令是 ./bin/kubernetes-session.sh \ -Dkubernetes.cluster-id=k8s-session-1 \ -Dtaskmanager.memory.process.size=4096m \ -Dkubernetes.taskmanager.cpu=2 \ -Dtaskmanager.numberOfTaskSlots=4 \ -Dresourcemanager.taskmanager-timeout=3600000 \ -Dkubernetes.container.image=flink:1.11.0-scala_2.11 但是会报错,找不到configmap 我看是执行上述命令是会生成configmap的,为什么还会报找不到。 |
Hi, zhai
图挂了。。可以整个图床工具贴出来,如果是异常直接贴文本也可以的。 Best, Leonard Xu > 在 2020年7月13日,19:59,Yvette zhai <[hidden email]> 写道: > > 大佬们,请教个问题,我是在k8s上部署flink1.11 natvie session模式。 > 下载的flink-1.11.0-bin-scala_2.11.tgz > > 执行命令是 > ./bin/kubernetes-session.sh \ > -Dkubernetes.cluster-id=k8s-session-1 \ > -Dtaskmanager.memory.process.size=4096m \ > -Dkubernetes.taskmanager.cpu=2 \ > -Dtaskmanager.numberOfTaskSlots=4 \ > -Dresourcemanager.taskmanager-timeout=3600000 \ > -Dkubernetes.container.image=flink:1.11.0-scala_2.11 > > 但是会报错,找不到configmap > > > 我看是执行上述命令是会生成configmap的,为什么还会报找不到。 > |
报错是MountVolume.SetUp failed for volume "flink-config-volume" : configmap
"flink-config-k8s-session-1" not found Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:03写道: > Hi, zhai > > 图挂了。。可以整个图床工具贴出来,如果是异常直接贴文本也可以的。 > > Best, > Leonard Xu > > > 在 2020年7月13日,19:59,Yvette zhai <[hidden email]> 写道: > > > > 大佬们,请教个问题,我是在k8s上部署flink1.11 natvie session模式。 > > 下载的flink-1.11.0-bin-scala_2.11.tgz > > > > 执行命令是 > > ./bin/kubernetes-session.sh \ > > -Dkubernetes.cluster-id=k8s-session-1 \ > > -Dtaskmanager.memory.process.size=4096m \ > > -Dkubernetes.taskmanager.cpu=2 \ > > -Dtaskmanager.numberOfTaskSlots=4 \ > > -Dresourcemanager.taskmanager-timeout=3600000 \ > > -Dkubernetes.container.image=flink:1.11.0-scala_2.11 > > > > 但是会报错,找不到configmap > > > > > > 我看是执行上述命令是会生成configmap的,为什么还会报找不到。 > > > > |
Hi, zhai
可以贴详细点吗?我帮你 CC 了熟悉这块的大佬 Yun Gao 祝好 > 在 2020年7月13日,20:11,Yvette zhai <[hidden email]> 写道: > > 报错是MountVolume.SetUp failed for volume "flink-config-volume" : configmap > "flink-config-k8s-session-1" not found > > > Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:03写道: > >> Hi, zhai >> >> 图挂了。。可以整个图床工具贴出来,如果是异常直接贴文本也可以的。 >> >> Best, >> Leonard Xu >> >>> 在 2020年7月13日,19:59,Yvette zhai <[hidden email]> 写道: >>> >>> 大佬们,请教个问题,我是在k8s上部署flink1.11 natvie session模式。 >>> 下载的flink-1.11.0-bin-scala_2.11.tgz >>> >>> 执行命令是 >>> ./bin/kubernetes-session.sh \ >>> -Dkubernetes.cluster-id=k8s-session-1 \ >>> -Dtaskmanager.memory.process.size=4096m \ >>> -Dkubernetes.taskmanager.cpu=2 \ >>> -Dtaskmanager.numberOfTaskSlots=4 \ >>> -Dresourcemanager.taskmanager-timeout=3600000 \ >>> -Dkubernetes.container.image=flink:1.11.0-scala_2.11 >>> >>> 但是会报错,找不到configmap >>> >>> >>> 我看是执行上述命令是会生成configmap的,为什么还会报找不到。 >>> >> >> |
1. 执行的脚本,产生的日志是:
2020-07-13 21:00:25,248 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost 2020-07-13 21:00:25,251 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123 2020-07-13 21:00:25,251 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m 2020-07-13 21:00:25,251 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m 2020-07-13 21:00:25,251 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1 2020-07-13 21:00:25,251 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1 2020-07-13 21:00:25,252 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region 2020-07-13 21:00:25,344 INFO org.apache.flink.client.deployment.DefaultClusterClientServiceLoader [] - Could not load factory due to missing dependencies. 2020-07-13 21:00:26,136 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead 2020-07-13 21:00:26,154 INFO org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment requires a fixed port. Configuration blob.server.port will be set to 6124 2020-07-13 21:00:26,154 INFO org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122 2020-07-13 21:00:26,204 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead 2020-07-13 21:00:26,220 WARN org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator [] - Found 0 files in directory null/etc/hadoop, skip to mount the Hadoop Configuration ConfigMap. 2020-07-13 21:00:26,220 WARN org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator [] - Found 0 files in directory null/etc/hadoop, skip to create the Hadoop Configuration ConfigMap. 2020-07-13 21:00:26,958 INFO org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Create flink session cluster k8s-session-1 successfully, JobManager Web Interface: http://172.16.5.175:8081 2. 查看 desrcibe 日志是: MountVolume.SetUp failed for volume "flink-config-volume" : configmap "flink-config-k8s-session-1" not found 3. logs 日志是: Start command : /bin/bash -c $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 -Dlog.file=/opt/flink/log/jobmanager.log -Dlogback.configurationFile=file:/opt/flink/conf/logback.xml -Dlog4j.configurationFile=file:/opt/flink/conf/log4j.properties org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint 1> /opt/flink/log/jobmanager.out 2> /opt/flink/log/jobmanager.err 4. kubectl get cm 可以看到 NAME DATA AGE flink-config-k8s-session-1 3 5m45s 麻烦大佬帮忙看看~是不是我的语句有问题还是缺什么文件~ 我是直接官网下的包,没有改任何文件~ Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:41写道: > Hi, zhai > > 可以贴详细点吗?我帮你 CC 了熟悉这块的大佬 Yun Gao > > 祝好 > > > 在 2020年7月13日,20:11,Yvette zhai <[hidden email]> 写道: > > > > 报错是MountVolume.SetUp failed for volume "flink-config-volume" : configmap > > "flink-config-k8s-session-1" not found > > > > > > Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:03写道: > > > >> Hi, zhai > >> > >> 图挂了。。可以整个图床工具贴出来,如果是异常直接贴文本也可以的。 > >> > >> Best, > >> Leonard Xu > >> > >>> 在 2020年7月13日,19:59,Yvette zhai <[hidden email]> 写道: > >>> > >>> 大佬们,请教个问题,我是在k8s上部署flink1.11 natvie session模式。 > >>> 下载的flink-1.11.0-bin-scala_2.11.tgz > >>> > >>> 执行命令是 > >>> ./bin/kubernetes-session.sh \ > >>> -Dkubernetes.cluster-id=k8s-session-1 \ > >>> -Dtaskmanager.memory.process.size=4096m \ > >>> -Dkubernetes.taskmanager.cpu=2 \ > >>> -Dtaskmanager.numberOfTaskSlots=4 \ > >>> -Dresourcemanager.taskmanager-timeout=3600000 \ > >>> -Dkubernetes.container.image=flink:1.11.0-scala_2.11 > >>> > >>> 但是会报错,找不到configmap > >>> > >>> > >>> 我看是执行上述命令是会生成configmap的,为什么还会报找不到。 > >>> > >> > >> > > |
补充一下,kubernetes版本是1.18
Yvette zhai <[hidden email]> 于2020年7月13日周一 下午9:10写道: > 1. 执行的脚本,产生的日志是: > 2020-07-13 21:00:25,248 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: jobmanager.rpc.address, localhost > 2020-07-13 21:00:25,251 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: jobmanager.rpc.port, 6123 > 2020-07-13 21:00:25,251 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: jobmanager.memory.process.size, 1600m > 2020-07-13 21:00:25,251 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: taskmanager.memory.process.size, 1728m > 2020-07-13 21:00:25,251 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: taskmanager.numberOfTaskSlots, 1 > 2020-07-13 21:00:25,251 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: parallelism.default, 1 > 2020-07-13 21:00:25,252 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: jobmanager.execution.failover-strategy, region > 2020-07-13 21:00:25,344 INFO > org.apache.flink.client.deployment.DefaultClusterClientServiceLoader [] - > Could not load factory due to missing dependencies. > 2020-07-13 21:00:26,136 INFO > org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The > derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is > less than its min value 192.000mb (201326592 bytes), min value will be used > instead > 2020-07-13 21:00:26,154 INFO > org.apache.flink.kubernetes.utils.KubernetesUtils [] - > Kubernetes deployment requires a fixed port. Configuration blob.server.port > will be set to 6124 > 2020-07-13 21:00:26,154 INFO > org.apache.flink.kubernetes.utils.KubernetesUtils [] - > Kubernetes deployment requires a fixed port. Configuration > taskmanager.rpc.port will be set to 6122 > 2020-07-13 21:00:26,204 INFO > org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The > derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is > less than its min value 192.000mb (201326592 bytes), min value will be used > instead > 2020-07-13 21:00:26,220 WARN > org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator > [] - Found 0 files in directory null/etc/hadoop, skip to mount the Hadoop > Configuration ConfigMap. > 2020-07-13 21:00:26,220 WARN > org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator > [] - Found 0 files in directory null/etc/hadoop, skip to create the Hadoop > Configuration ConfigMap. > 2020-07-13 21:00:26,958 INFO > org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Create > flink session cluster k8s-session-1 successfully, JobManager Web Interface: > http://172.16.5.175:8081 > > 2. 查看 desrcibe 日志是: > MountVolume.SetUp failed for volume "flink-config-volume" : configmap > "flink-config-k8s-session-1" not found > > 3. logs 日志是: > > Start command : /bin/bash -c $JAVA_HOME/bin/java -classpath > $FLINK_CLASSPATH -Xmx1073741824 -Xms1073741824 > -XX:MaxMetaspaceSize=268435456 -Dlog.file=/opt/flink/log/jobmanager.log > -Dlogback.configurationFile=file:/opt/flink/conf/logback.xml > -Dlog4j.configurationFile=file:/opt/flink/conf/log4j.properties > org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint > 1> /opt/flink/log/jobmanager.out 2> /opt/flink/log/jobmanager.err > > 4. kubectl get cm 可以看到 > NAME DATA AGE > flink-config-k8s-session-1 3 5m45s > > 麻烦大佬帮忙看看~是不是我的语句有问题还是缺什么文件~ > 我是直接官网下的包,没有改任何文件~ > > Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:41写道: > >> Hi, zhai >> >> 可以贴详细点吗?我帮你 CC 了熟悉这块的大佬 Yun Gao >> >> 祝好 >> >> > 在 2020年7月13日,20:11,Yvette zhai <[hidden email]> 写道: >> > >> > 报错是MountVolume.SetUp failed for volume "flink-config-volume" : configmap >> > "flink-config-k8s-session-1" not found >> > >> > >> > Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:03写道: >> > >> >> Hi, zhai >> >> >> >> 图挂了。。可以整个图床工具贴出来,如果是异常直接贴文本也可以的。 >> >> >> >> Best, >> >> Leonard Xu >> >> >> >>> 在 2020年7月13日,19:59,Yvette zhai <[hidden email]> 写道: >> >>> >> >>> 大佬们,请教个问题,我是在k8s上部署flink1.11 natvie session模式。 >> >>> 下载的flink-1.11.0-bin-scala_2.11.tgz >> >>> >> >>> 执行命令是 >> >>> ./bin/kubernetes-session.sh \ >> >>> -Dkubernetes.cluster-id=k8s-session-1 \ >> >>> -Dtaskmanager.memory.process.size=4096m \ >> >>> -Dkubernetes.taskmanager.cpu=2 \ >> >>> -Dtaskmanager.numberOfTaskSlots=4 \ >> >>> -Dresourcemanager.taskmanager-timeout=3600000 \ >> >>> -Dkubernetes.container.image=flink:1.11.0-scala_2.11 >> >>> >> >>> 但是会报错,找不到configmap >> >>> >> >>> >> >>> 我看是执行上述命令是会生成configmap的,为什么还会报找不到。 >> >>> >> >> >> >> >> >> |
configmap "flink-config-k8s-session-1" not found的报错是正常的
因为目前的实现是先创建JobManager Deployment,然后再创建ConfigMap并设置owner reference到deployment 所以你才会看到创建Pod的时候报ConfigMap还没有创建出来,这个是正常的信息,K8s会自动重试创建Pod 你现在是任务起不来吗,还是有什么其他的问题? Best, Yang Yvette zhai <[hidden email]> 于2020年7月14日周二 上午10:20写道: > 补充一下,kubernetes版本是1.18 > Yvette zhai <[hidden email]> 于2020年7月13日周一 下午9:10写道: > > > 1. 执行的脚本,产生的日志是: > > 2020-07-13 21:00:25,248 INFO > > org.apache.flink.configuration.GlobalConfiguration [] - > Loading > > configuration property: jobmanager.rpc.address, localhost > > 2020-07-13 21:00:25,251 INFO > > org.apache.flink.configuration.GlobalConfiguration [] - > Loading > > configuration property: jobmanager.rpc.port, 6123 > > 2020-07-13 21:00:25,251 INFO > > org.apache.flink.configuration.GlobalConfiguration [] - > Loading > > configuration property: jobmanager.memory.process.size, 1600m > > 2020-07-13 21:00:25,251 INFO > > org.apache.flink.configuration.GlobalConfiguration [] - > Loading > > configuration property: taskmanager.memory.process.size, 1728m > > 2020-07-13 21:00:25,251 INFO > > org.apache.flink.configuration.GlobalConfiguration [] - > Loading > > configuration property: taskmanager.numberOfTaskSlots, 1 > > 2020-07-13 21:00:25,251 INFO > > org.apache.flink.configuration.GlobalConfiguration [] - > Loading > > configuration property: parallelism.default, 1 > > 2020-07-13 21:00:25,252 INFO > > org.apache.flink.configuration.GlobalConfiguration [] - > Loading > > configuration property: jobmanager.execution.failover-strategy, region > > 2020-07-13 21:00:25,344 INFO > > org.apache.flink.client.deployment.DefaultClusterClientServiceLoader [] > - > > Could not load factory due to missing dependencies. > > 2020-07-13 21:00:26,136 INFO > > org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The > > derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) > is > > less than its min value 192.000mb (201326592 bytes), min value will be > used > > instead > > 2020-07-13 21:00:26,154 INFO > > org.apache.flink.kubernetes.utils.KubernetesUtils [] - > > Kubernetes deployment requires a fixed port. Configuration > blob.server.port > > will be set to 6124 > > 2020-07-13 21:00:26,154 INFO > > org.apache.flink.kubernetes.utils.KubernetesUtils [] - > > Kubernetes deployment requires a fixed port. Configuration > > taskmanager.rpc.port will be set to 6122 > > 2020-07-13 21:00:26,204 INFO > > org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The > > derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) > is > > less than its min value 192.000mb (201326592 bytes), min value will be > used > > instead > > 2020-07-13 21:00:26,220 WARN > > > org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator > > [] - Found 0 files in directory null/etc/hadoop, skip to mount the Hadoop > > Configuration ConfigMap. > > 2020-07-13 21:00:26,220 WARN > > > org.apache.flink.kubernetes.kubeclient.decorators.HadoopConfMountDecorator > > [] - Found 0 files in directory null/etc/hadoop, skip to create the > Hadoop > > Configuration ConfigMap. > > 2020-07-13 21:00:26,958 INFO > > org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Create > > flink session cluster k8s-session-1 successfully, JobManager Web > Interface: > > http://172.16.5.175:8081 > > > > 2. 查看 desrcibe 日志是: > > MountVolume.SetUp failed for volume "flink-config-volume" : configmap > > "flink-config-k8s-session-1" not found > > > > 3. logs 日志是: > > > > Start command : /bin/bash -c $JAVA_HOME/bin/java -classpath > > $FLINK_CLASSPATH -Xmx1073741824 -Xms1073741824 > > -XX:MaxMetaspaceSize=268435456 -Dlog.file=/opt/flink/log/jobmanager.log > > -Dlogback.configurationFile=file:/opt/flink/conf/logback.xml > > -Dlog4j.configurationFile=file:/opt/flink/conf/log4j.properties > > org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint > > 1> /opt/flink/log/jobmanager.out 2> /opt/flink/log/jobmanager.err > > > > 4. kubectl get cm 可以看到 > > NAME DATA AGE > > flink-config-k8s-session-1 3 5m45s > > > > 麻烦大佬帮忙看看~是不是我的语句有问题还是缺什么文件~ > > 我是直接官网下的包,没有改任何文件~ > > > > Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:41写道: > > > >> Hi, zhai > >> > >> 可以贴详细点吗?我帮你 CC 了熟悉这块的大佬 Yun Gao > >> > >> 祝好 > >> > >> > 在 2020年7月13日,20:11,Yvette zhai <[hidden email]> 写道: > >> > > >> > 报错是MountVolume.SetUp failed for volume "flink-config-volume" : > configmap > >> > "flink-config-k8s-session-1" not found > >> > > >> > > >> > Leonard Xu <[hidden email]> 于2020年7月13日周一 下午8:03写道: > >> > > >> >> Hi, zhai > >> >> > >> >> 图挂了。。可以整个图床工具贴出来,如果是异常直接贴文本也可以的。 > >> >> > >> >> Best, > >> >> Leonard Xu > >> >> > >> >>> 在 2020年7月13日,19:59,Yvette zhai <[hidden email]> 写道: > >> >>> > >> >>> 大佬们,请教个问题,我是在k8s上部署flink1.11 natvie session模式。 > >> >>> 下载的flink-1.11.0-bin-scala_2.11.tgz > >> >>> > >> >>> 执行命令是 > >> >>> ./bin/kubernetes-session.sh \ > >> >>> -Dkubernetes.cluster-id=k8s-session-1 \ > >> >>> -Dtaskmanager.memory.process.size=4096m \ > >> >>> -Dkubernetes.taskmanager.cpu=2 \ > >> >>> -Dtaskmanager.numberOfTaskSlots=4 \ > >> >>> -Dresourcemanager.taskmanager-timeout=3600000 \ > >> >>> -Dkubernetes.container.image=flink:1.11.0-scala_2.11 > >> >>> > >> >>> 但是会报错,找不到configmap > >> >>> > >> >>> > >> >>> 我看是执行上述命令是会生成configmap的,为什么还会报找不到。 > >> >>> > >> >> > >> >> > >> > >> > |
您好,我也遇到了同样的问题。
MountVolume.SetUp failed for volume "flink-config-volume" : configmap "flink-config-flink-mm" not found Back-off restarting failed container 查看对应namespace 下的configmap,flink-config-flink-mm已经存在。但是JobManager pod 还是一直在重试,不能提供服务。 -- Sent from: http://apache-flink.147419.n8.nabble.com/ |
失败的根本原因应该不是ConfigMap找不到,warning的那个信息是因为创建JobManager deployment的时候
ConfigMap还没创建出来,不会导致失败的。 你可以参考这个地方[1]把JobManager的log的打到console里面,然后用kubectl logs <podName>来查看,这样 可以排查JobManager一直crash backoff的原因 [1]. https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#log-files Best, Yang Fy <[hidden email]> 于2020年11月4日周三 下午5:36写道: > 您好,我也遇到了同样的问题。 > MountVolume.SetUp failed for volume "flink-config-volume" : configmap > "flink-config-flink-mm" not found > Back-off restarting failed container > 查看对应namespace 下的configmap,flink-config-flink-mm已经存在。但是JobManager pod > 还是一直在重试,不能提供服务。 > > > > -- > Sent from: http://apache-flink.147419.n8.nabble.com/ > |
Free forum by Nabble | Edit this page |