关于flink 提交job参数不生效的问题

[guanyq]

flink 提交jar包是 指定-ytm不起作用。想知道什么原因？

 

[Xintong Song]

你邮件里的图片没有显示出来。
建议把完整的启动命令贴一下。

Thank you~

Xintong Song



On Tue, Apr 14, 2020 at 1:11 PM guanyq <[hidden email]> wrote:

> flink 提交jar包是 指定-ytm不起作用。想知道什么原因？
>
>
>
>

[guanyq]

./bin/flink run -m yarn-cluster \<br/>-ynm TestDataProcess \<br/>-ytm 666 \<br/>-yjm 666 \<br/>-c com.data.processing.unconditionalacceptance.TestDataProcess \<br/>./tasks/UnconditionalAcceptanceDataProcess.jar \<br/>--group.id Test001 \<br/>--checkpoint.interval 5000
在 2020-04-14 14:00:59，"Xintong Song" <[hidden email]> 写道：

>你邮件里的图片没有显示出来。
>建议把完整的启动命令贴一下。
>
>Thank you~
>
>Xintong Song
>
>
>
>On Tue, Apr 14, 2020 at 1:11 PM guanyq <[hidden email]> wrote:
>
>> flink 提交jar包是 指定-ytm不起作用。想知道什么原因？
>>
>>
>>
>>

[Xintong Song]

启动命令看起来是对的。
你说的不起作用，具体是什么现象呢？

Thank you~

Xintong Song



On Tue, Apr 14, 2020 at 2:05 PM guanyq <[hidden email]> wrote:

> ./bin/flink run -m yarn-cluster \<br/>-ynm TestDataProcess \<br/>-ytm 666
> \<br/>-yjm 666 \<br/>-c
> com.data.processing.unconditionalacceptance.TestDataProcess
> \<br/>./tasks/UnconditionalAcceptanceDataProcess.jar \<br/>--group.id
> Test001 \<br/>--checkpoint.interval 5000
> 在 2020-04-14 14:00:59，"Xintong Song" <[hidden email]> 写道：
> >你邮件里的图片没有显示出来。
> >建议把完整的启动命令贴一下。
> >
> >Thank you~
> >
> >Xintong Song
> >
> >
> >
> >On Tue, Apr 14, 2020 at 1:11 PM guanyq <[hidden email]> wrote:
> >
> >> flink 提交jar包是 指定-ytm不起作用。想知道什么原因？
> >>
> >>
> >>
> >>
>

[guanyq]

脚本设置-ytm 666<br/>但是flink ui页面的，job manager--&gt;taskmanager.heap.size为1024
在 2020-04-14 14:10:31，"Xintong Song" <[hidden email]> 写道：

>启动命令看起来是对的。
>你说的不起作用，具体是什么现象呢？
>
>Thank you~
>
>Xintong Song
>
>
>
>On Tue, Apr 14, 2020 at 2:05 PM guanyq <[hidden email]> wrote:
>
>> ./bin/flink run -m yarn-cluster \<br/>-ynm TestDataProcess \<br/>-ytm 666
>> \<br/>-yjm 666 \<br/>-c
>> com.data.processing.unconditionalacceptance.TestDataProcess
>> \<br/>./tasks/UnconditionalAcceptanceDataProcess.jar \<br/>--group.id
>> Test001 \<br/>--checkpoint.interval 5000
>> 在 2020-04-14 14:00:59，"Xintong Song" <[hidden email]> 写道：
>> >你邮件里的图片没有显示出来。
>> >建议把完整的启动命令贴一下。
>> >
>> >Thank you~
>> >
>> >Xintong Song
>> >
>> >
>> >
>> >On Tue, Apr 14, 2020 at 1:11 PM guanyq <[hidden email]> wrote:
>> >
>> >> flink 提交jar包是 指定-ytm不起作用。想知道什么原因？
>> >>
>> >>
>> >>
>> >>
>>

  应该是你设置的 -ytm 和 -yjm内存大小比yarn container最小容器内存都小吧！
 yarn最小容器内存的参数： yarn.scheduler.minimum-allocation-mb
容器内存增量： yarn.scheduler.increment-allocation-mb

发件人： guanyq
发送时间： 2020-04-14 14:05
收件人： user-zh
主题： Re:Re: 关于flink 提交job参数不生效的问题
./bin/flink run -m yarn-cluster \<br/>-ynm TestDataProcess \<br/>-ytm 666 \<br/>-yjm 666 \<br/>-c com.data.processing.unconditionalacceptance.TestDataProcess \<br/>./tasks/UnconditionalAcceptanceDataProcess.jar \<br/>--group.id Test001 \<br/>--checkpoint.interval 5000
在 2020-04-14 14:00:59，"Xintong Song" <[hidden email]> 写道：

>你邮件里的图片没有显示出来。
>建议把完整的启动命令贴一下。
>
>Thank you~
>
>Xintong Song
>
>
>
>On Tue, Apr 14, 2020 at 1:11 PM guanyq <[hidden email]> wrote:
>
>> flink 提交jar包是 指定-ytm不起作用。想知道什么原因？
>>
>>
>>
>>

[zhisheng]

可以试试设置 -ytm 2048m  看看是不是还这样

[hidden email] <[hidden email]> 于2020年4月14日周二
下午2:16写道：

>
>   应该是你设置的 -ytm 和 -yjm内存大小比yarn container最小容器内存都小吧！
>  yarn最小容器内存的参数： yarn.scheduler.minimum-allocation-mb
> 容器内存增量： yarn.scheduler.increment-allocation-mb
>
> 发件人： guanyq
> 发送时间： 2020-04-14 14:05
> 收件人： user-zh
> 主题： Re:Re: 关于flink 提交job参数不生效的问题
> ./bin/flink run -m yarn-cluster \<br/>-ynm TestDataProcess \<br/>-ytm 666
> \<br/>-yjm 666 \<br/>-c
> com.data.processing.unconditionalacceptance.TestDataProcess
> \<br/>./tasks/UnconditionalAcceptanceDataProcess.jar \<br/>--group.id
> Test001 \<br/>--checkpoint.interval 5000
> 在 2020-04-14 14:00:59，"Xintong Song" <[hidden email]> 写道：
> >你邮件里的图片没有显示出来。
> >建议把完整的启动命令贴一下。
> >
> >Thank you~
> >
> >Xintong Song
> >
> >
> >
> >On Tue, Apr 14, 2020 at 1:11 PM guanyq <[hidden email]> wrote:
> >
> >> flink 提交jar包是 指定-ytm不起作用。想知道什么原因？
> >>
> >>
> >>
> >>
>

[guanyq]

提交失败，yarn资源也还有很多，为什么会提交失败呢？

提交脚本
./bin/flink run -m yarn-cluster \
-ynm TestDataProcess \
-yd \
-yn 2 \
-ytm 1024 \
-yjm 1024 \
-c com.data.processing.unconditionalacceptance.TestDataProcess \
./tasks/UnconditionalAcceptanceDataProcess.jar \


yarn资源
Apps Submitted Apps Pending Apps Running Apps Completed Containers Running Memory Used Memory Total Memory Reserved VCores Used VCores Total VCores Reserved Active Nodes Decommissioned Nodes Lost Nodes Unhealthy Nodes Rebooted Nodes
239 0 12 227 173 334 GB 1.42 TB 0 B 173 288 0 9 0 0 0 0



2020-04-14 15:14:19,002 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:19,253 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:19,504 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:19,755 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:20,006 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:20,257 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:20,508 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:20,759 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:21,011 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:21,262 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:21,513 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:21,764 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:22,015 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:22,265 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:22,517 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:22,768 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:23,019 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:23,270 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:23,521 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:23,772 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:24,025 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2020-04-14 15:14:24,276 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

[guanyq]

提交失败，本人测试与-yd参数有关系，<br/>这个参数去掉就可以提交了。<br/>但是不知道 -yd这个参数影响了什么？
At 2020-04-14 15:31:00, "guanyq" <[hidden email]> wrote:

>提交失败，yarn资源也还有很多，为什么会提交失败呢？
>
>提交脚本
>./bin/flink run -m yarn-cluster \
>-ynm TestDataProcess \
>-yd \
>-yn 2 \
>-ytm 1024 \
>-yjm 1024 \
>-c com.data.processing.unconditionalacceptance.TestDataProcess \
>./tasks/UnconditionalAcceptanceDataProcess.jar \
>
>
>yarn资源
>Apps Submitted Apps Pending Apps Running Apps Completed Containers Running Memory Used Memory Total Memory Reserved VCores Used VCores Total VCores Reserved Active Nodes Decommissioned Nodes Lost Nodes Unhealthy Nodes Rebooted Nodes
>239 0 12 227 173 334 GB 1.42 TB 0 B 173 288 0 9 0 0 0 0
>
>
>
>2020-04-14 15:14:19,002 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:19,253 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:19,504 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:19,755 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:20,006 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:20,257 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:20,508 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:20,759 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:21,011 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:21,262 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:21,513 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:21,764 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:22,015 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:22,265 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:22,517 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:22,768 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:23,019 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:23,270 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:23,521 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:23,772 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:24,025 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
>2020-04-14 15:14:24,276 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

[tison]

-yd 参数影响的是你是否使用 perjob 模式提交作业，简单地说

with -yd 以 perjob 模式提交作业，即启动一个新集群
without -yd 提交到一个现有的 Flink on YARN 集群

哪个是你的需求呢？有没有实现用 yarn-session 启动 Flink on YARN 集群呢？

Best,
tison.


guanyq <[hidden email]> 于2020年4月15日周三 上午8:46写道：

> 提交失败，本人测试与-yd参数有关系，<br/>这个参数去掉就可以提交了。<br/>但是不知道 -yd这个参数影响了什么？
> At 2020-04-14 15:31:00, "guanyq" <[hidden email]> wrote:
> >提交失败，yarn资源也还有很多，为什么会提交失败呢？
> >
> >提交脚本
> >./bin/flink run -m yarn-cluster \
> >-ynm TestDataProcess \
> >-yd \
> >-yn 2 \
> >-ytm 1024 \
> >-yjm 1024 \
> >-c com.data.processing.unconditionalacceptance.TestDataProcess \
> >./tasks/UnconditionalAcceptanceDataProcess.jar \
> >
> >
> >yarn资源
> >Apps Submitted Apps Pending    Apps Running    Apps Completed  Containers
> Running      Memory Used     Memory Total    Memory Reserved VCores Used
>  VCores Total    VCores Reserved Active Nodes    Decommissioned Nodes
> Lost Nodes      Unhealthy Nodes Rebooted Nodes
> >239    0       12      227     173     334 GB  1.42 TB 0 B     173
>  288     0       9       0       0       0       0
> >
> >
> >
> >2020-04-14 15:14:19,002 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:19,253 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:19,504 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:19,755 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:20,006 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:20,257 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:20,508 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:20,759 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:21,011 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:21,262 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:21,513 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:21,764 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:22,015 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:22,265 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:22,517 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:22,768 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:23,019 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:23,270 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:23,521 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:23,772 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:24,025 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
> >2020-04-14 15:14:24,276 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
> took more than 60 seconds. Please check if the requested resources are
> available in the YARN cluster
>

[guanyq]

使用的是perjob模式提交作业，没有使用yarn-seesion。为什么perjob模式提交有这个-yd参数会有问题，还是没太懂。
在 2020-04-15 08:52:11，"tison" <[hidden email]> 写道：

>-yd 参数影响的是你是否使用 perjob 模式提交作业，简单地说
>
>with -yd 以 perjob 模式提交作业，即启动一个新集群
>without -yd 提交到一个现有的 Flink on YARN 集群
>
>哪个是你的需求呢？有没有实现用 yarn-session 启动 Flink on YARN 集群呢？
>
>Best,
>tison.
>
>
>guanyq <[hidden email]> 于2020年4月15日周三 上午8:46写道：
>
>> 提交失败，本人测试与-yd参数有关系，<br/>这个参数去掉就可以提交了。<br/>但是不知道 -yd这个参数影响了什么？
>> At 2020-04-14 15:31:00, "guanyq" <[hidden email]> wrote:
>> >提交失败，yarn资源也还有很多，为什么会提交失败呢？
>> >
>> >提交脚本
>> >./bin/flink run -m yarn-cluster \
>> >-ynm TestDataProcess \
>> >-yd \
>> >-yn 2 \
>> >-ytm 1024 \
>> >-yjm 1024 \
>> >-c com.data.processing.unconditionalacceptance.TestDataProcess \
>> >./tasks/UnconditionalAcceptanceDataProcess.jar \
>> >
>> >
>> >yarn资源
>> >Apps Submitted Apps Pending    Apps Running    Apps Completed  Containers
>> Running      Memory Used     Memory Total    Memory Reserved VCores Used
>>  VCores Total    VCores Reserved Active Nodes    Decommissioned Nodes
>> Lost Nodes      Unhealthy Nodes Rebooted Nodes
>> >239    0       12      227     173     334 GB  1.42 TB 0 B     173
>>  288     0       9       0       0       0       0
>> >
>> >
>> >
>> >2020-04-14 15:14:19,002 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:19,253 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:19,504 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:19,755 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:20,006 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:20,257 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:20,508 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:20,759 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:21,011 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:21,262 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:21,513 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:21,764 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:22,015 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:22,265 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:22,517 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:22,768 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:23,019 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:23,270 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:23,521 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:23,772 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:24,025 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>> >2020-04-14 15:14:24,276 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>> took more than 60 seconds. Please check if the requested resources are
>> available in the YARN cluster
>>

[guanyq]

代码里面是有env.execute，提交job出现以下错误，可能时什么原因？
The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.

[guanyq]

ok 找到原因了！不好意思！
在 2020-04-16 08:03:29，"guanyq" <[hidden email]> 写道：
>代码里面是有env.execute，提交job出现以下错误，可能时什么原因？
>The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.

[Weihua Hu]

Hi, dlguanyq

Deployment took more than 60 seconds.
——
这个日志表示已经将 Application 提交到 Yarn 了，但是 AM 一直没有启动，这一步和perjob模式或者 yarn-session 模式关系不大。
可以用 -yd 多试几次，还是不能成功的话，需要检查下 yarn 的日志

Best
Weihua Hu

> 2020年4月15日 09:06，guanyq <[hidden email]> 写道：
>
> 使用的是perjob模式提交作业，没有使用yarn-seesion。为什么perjob模式提交有这个-yd参数会有问题，还是没太懂。
> 在 2020-04-15 08:52:11，"tison" <[hidden email]> 写道：
>> -yd 参数影响的是你是否使用 perjob 模式提交作业，简单地说
>>
>> with -yd 以 perjob 模式提交作业，即启动一个新集群
>> without -yd 提交到一个现有的 Flink on YARN 集群
>>
>> 哪个是你的需求呢？有没有实现用 yarn-session 启动 Flink on YARN 集群呢？
>>
>> Best,
>> tison.
>>
>>
>> guanyq <[hidden email]> 于2020年4月15日周三 上午8:46写道：
>>
>>> 提交失败，本人测试与-yd参数有关系，<br/>这个参数去掉就可以提交了。<br/>但是不知道 -yd这个参数影响了什么？
>>> At 2020-04-14 15:31:00, "guanyq" <[hidden email]> wrote:
>>>> 提交失败，yarn资源也还有很多，为什么会提交失败呢？
>>>>
>>>> 提交脚本
>>>> ./bin/flink run -m yarn-cluster \
>>>> -ynm TestDataProcess \
>>>> -yd \
>>>> -yn 2 \
>>>> -ytm 1024 \
>>>> -yjm 1024 \
>>>> -c com.data.processing.unconditionalacceptance.TestDataProcess \
>>>> ./tasks/UnconditionalAcceptanceDataProcess.jar \
>>>>
>>>>
>>>> yarn资源
>>>> Apps Submitted Apps Pending    Apps Running    Apps Completed  Containers
>>> Running      Memory Used     Memory Total    Memory Reserved VCores Used
>>> VCores Total    VCores Reserved Active Nodes    Decommissioned Nodes
>>> Lost Nodes      Unhealthy Nodes Rebooted Nodes
>>>> 239    0       12      227     173     334 GB  1.42 TB 0 B     173
>>> 288     0       9       0       0       0       0
>>>>
>>>>
>>>>
>>>> 2020-04-14 15:14:19,002 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:19,253 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:19,504 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:19,755 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:20,006 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:20,257 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:20,508 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:20,759 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:21,011 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:21,262 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:21,513 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:21,764 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:22,015 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:22,265 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:22,517 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:22,768 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:23,019 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:23,270 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:23,521 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:23,772 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:24,025 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>> 2020-04-14 15:14:24,276 INFO
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deployment
>>> took more than 60 seconds. Please check if the requested resources are
>>> available in the YARN cluster
>>>

[guanyq]

ok 找到原因了！不好意思！
在 2020-04-16 08:03:29，"guanyq" <[hidden email]> 写道：
>代码里面是有env.execute，提交job出现以下错误，可能时什么原因？
>The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.

[guanyq]

附件是代码和错误日志。目前不知道如何调查。麻烦帮忙看下 谢谢。

[Congxian Qiu]

Hi
从错误日志看，是 StateMigration 相关的问题。
你需要确认下，你的代码中的 Serializer 和 savepoint 中 state 相关的 serializer
是一样的或者是兼容的，你可以参考下这个文档[1]

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/schema_evolution.html

Best,
Congxian


guanyq <[hidden email]> 于2020年4月29日周三 下午6:09写道：

>
> 附件是代码和错误日志。目前不知道如何调查。麻烦帮忙看下 谢谢。

[guanyq]

代码中没特别指定Serializer。都是默认的序列化。
在 2020-04-29 18:20:22，"Congxian Qiu" <[hidden email]> 写道：

>Hi
>从错误日志看，是 StateMigration 相关的问题。
>你需要确认下，你的代码中的 Serializer 和 savepoint 中 state 相关的 serializer
>是一样的或者是兼容的，你可以参考下这个文档[1]
>
>[1]
>https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/schema_evolution.html
>
>Best,
>Congxian
>
>
>guanyq <[hidden email]> 于2020年4月29日周三 下午6:09写道：
>
>>
>> 附件是代码和错误日志。目前不知道如何调查。麻烦帮忙看下 谢谢。

[shao.hongxiao]

能发一下写入状态的代码看一下吗，还有一个问题，键值状态访问，你的代码里是读出了所有key关键的mapstate吗，谢谢




| |
邵红晓
|
|
邮箱：[hidden email]
|

签名由 网易邮箱大师 定制

在2020年04月30日 09:04，guanyq 写道：
代码中没特别指定Serializer。都是默认的序列化。
在 2020-04-29 18:20:22，"Congxian Qiu" <[hidden email]> 写道：

>Hi
>从错误日志看，是 StateMigration 相关的问题。
>你需要确认下，你的代码中的 Serializer 和 savepoint 中 state 相关的 serializer
>是一样的或者是兼容的，你可以参考下这个文档[1]
>
>[1]
>https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/schema_evolution.html
>
>Best,
>Congxian
>
>
>guanyq <[hidden email]> 于2020年4月29日周三 下午6:09写道：
>
>>
>> 附件是代码和错误日志。目前不知道如何调查。麻烦帮忙看下 谢谢。

[guanyq]

附件是代码

还有一个问题，键值状态访问，你的代码里是读出了所有key关键的mapstate吗

-- 代码是读出所有map状态的key。

在 2020-04-30 09:40:45，"shx" <[hidden email]> 写道：
>能发一下写入状态的代码看一下吗，还有一个问题，键值状态访问，你的代码里是读出了所有key关键的mapstate吗，谢谢
>
>
>
>
>| |
>邵红晓
>|
>|
>邮箱：[hidden email]
>|
>
>签名由 网易邮箱大师 定制
>
>在2020年04月30日 09:04，guanyq 写道：
>代码中没特别指定Serializer。都是默认的序列化。
>在 2020-04-29 18:20:22，"Congxian Qiu" <[hidden email]> 写道：
>>Hi
>>从错误日志看，是 StateMigration 相关的问题。
>>你需要确认下，你的代码中的 Serializer 和 savepoint 中 state 相关的 serializer
>>是一样的或者是兼容的，你可以参考下这个文档[1]
>>
>>[1]
>>https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/schema_evolution.html
>>
>>Best,
>>Congxian
>>
>>
>>guanyq <[hidden email]> 于2020年4月29日周三 下午6:09写道：
>>
>>>
>>> 附件是代码和错误日志。目前不知道如何调查。麻烦帮忙看下 谢谢。