Re:flink 1.11.0 chk超时

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re:flink 1.11.0 chk超时

hailongwang
Hi liangji,


CP 超时的原因一般是因任务而议的。从你提供的 2 张截图来看,卡在第二个 operator 的 subtask3 上。
上下两个 operator 之间的关系是 forworad 还是 reblance 呢?如果是 forward 的话,可以看下是不是数据倾斜,subtask3 需要处理的数据量比较多。
如果是 reblance 的话,以为 subtask1 和 subtask2 都成功了,所以上游的 barrier 应该都往下发了,所以 subtask3也收到了上游的 barrier,而 reblance 数据量都一样,所以可以看下是不是 sink 出去太慢导致。
查看任务一般可以看下任务的日志, GC,采堆栈,画火焰图等。


Best,
Hailong Wang


在 2020-11-05 16:26:06,"liangji" <[hidden email]> 写道:

><http://apache-flink.147419.n8.nabble.com/file/t538/QQ%E6%88%AA%E5%9B%BE20201105165123.jpg>
><http://apache-flink.147419.n8.nabble.com/file/t538/QQ%E6%88%AA%E5%9B%BE20201105165200.jpg>
>
>chk history如图,以下是TM中找到的INFO信息:
>2020-11-05 13:13:38,101 INFO
>org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] -
>Subtask 2 checkpointing for checkpoint with id=16 (max part counter=6).
>2020-11-05 13:13:38,143 INFO
>org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] -
>Subtask 1 checkpointing for checkpoint with id=16 (max part counter=0).
>2020-11-05 13:14:37,779 WARN  org.apache.kafka.clients.NetworkClient                      
>[] - Connection to node -3 could not be established. Broker may not be
>available.
>2020-11-05 13:14:37,786 WARN  org.apache.kafka.clients.NetworkClient                      
>[] - Connection to node -2 could not be established. Broker may not be
>available.
>2020-11-05 13:33:38,115 INFO  org.apache.flink.runtime.taskmanager.Task                  
>[] - Attempting to cancel task Source: Custom Source -> Process -> (Sink:
>Hdfs sink, Sink: HistoryTopic hbase sink, Sink: HistoryTopicCopy hbase sink)
>(3/3) (68bfa6305a9aa5a7381b9ca4a8fef2fa).
>请路过的大佬们指点下chk超时怎么定位问题,多谢(目前日志级别切换不成debug)
>
>
>
>--
>Sent from: http://apache-flink.147419.n8.nabble.com/