flink 执行一段时间后报错：Requesting TaskManager's path for query services failed.

[wangsong2]

flink 版本：1.9.0

大家好，flink任务在执行一段时间后，出现报错：Requesting TaskManager's path for query services
failed，akka.pattern.AskTimeoutException，请问这个原因是什么呢？详细错误输出如下：

2020-06-15 02:12:07,434 WARN
org.apache.flink.runtime.rest.handler.legacy.metrics.MetricFetcherImpl -
Requesting TaskManager's path for query services failed.
java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException:
Ask timed out on [Actor[akka://flink/user/dispatcher#-1311898795]] after
[10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply.
    at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
    at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
    at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
    at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
    at
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:871)
    at akka.dispatch.OnComplete.internal(Future.scala:263)
    at akka.dispatch.OnComplete.internal(Future.scala:261)
    at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
    at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
    at
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74)
    at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
    at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
    at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:644)
    at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
    at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
    at
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
    at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
    at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
    at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
    at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
    at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
    at java.lang.Thread.run(Thread.java:748)
Caused by: akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#-1311898795]] after [10000 ms]. Message
of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A
typical reason for `AskTimeoutException` is that the recipient actor didn't
send a reply.
    at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
    at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
    at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
    ... 9 more
2020-06-15 02:12:12,603 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (2/32) (689c6e191be63afea5182581e2875d4d) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,604 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (4/32) (d2f0fbf94e3d7cddc0f20080f94a1692) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,604 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (3/32) (28c8833ebbc14377bc74e7a389b4220b) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,604 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (5/32) (585854caaef14df77b084f92eb5fb4a2) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,604 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (15/32) (fc699114e90473fda9c717b5e0d4b716) switched from CANCELING
to CANCELED.
2020-06-15 02:12:12,605 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (22/32) (e3569b88e95af3d06e751ade5cdde34c) switched from CANCELING
to CANCELED.
2020-06-15 02:12:12,605 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (31/32) (a653c6cbb83c04b6c3f36a3ba80e3979) switched from CANCELING
to CANCELED.
2020-06-15 02:12:12,605 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter ->
Map (1/6) (89af922e79d370e2df9fb4a9629ad100) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,606 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter ->
Map (2/6) (7006987f6159a65598fe11237490a666) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,606 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter ->
Map (3/6) (fd709c1ff1d68e00809083bea34d0f2c) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,606 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter ->
Map (4/6) (8315275277f09302c830f7d9c9bea26f) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,607 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter ->
Map (5/6) (6a43b3397918a378a1ff54e9c8b81ca1) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,607 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter ->
Map (6/6) (955889cf6013c1d379a82eebd445c9a6) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,607 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Map (1/32) (05ff6ce269ef904fe379b66ba715d704) switched from CANCELING to
CANCELED.
2020-06-15 02:12:12,614 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Sink: prm (1/3) (7093e2d773dadfa4de49925cc0a85c8a) switched from
CANCELING to CANCELED.
2020-06-15 02:12:12,625 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Sink: prm (2/3) (e0ca0d779887e69e79a78ec9e1668cda) switched from
CANCELING to CANCELED.
2020-06-15 02:12:12,629 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
Co-Process-Broadcast -> Sink: tget (3/3) (0cb1f3bf90fa14c94a36dfe6f5808e8c)
switched from CANCELING to CANCELED.
2020-06-15 02:12:12,629 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
Co-Process-Broadcast -> Sink: tget (1/3) (e59555e50c1d00c68a28a9c1e130ee26)
switched from CANCELING to CANCELED.
2020-06-15 02:12:12,630 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map -> Filter
-> Sink: prm (3/3) (3bac2e02e88a9a3b7d3333a73566e36b) switched from
CANCELING to CANCELED.
2020-06-15 02:12:12,774 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
Co-Process-Broadcast -> Sink: tk (2/4) (184d88d91ea62eb0b5c2b8660cb3b655)
switched from CANCELING to CANCELED.
2020-06-15 02:12:12,775 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
Co-Process-Broadcast -> Sink: tk (4/4) (eb576b62cc4cd6e5e3f55de4c0122f4a)
switched from CANCELING to CANCELED.
2020-06-15 02:12:13,315 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom
Source (1/1) (039627fde72d22438dea1485c69d379b) switched from CANCELING to
CANCELED.
2020-06-15 02:12:14,536 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom
Source -> Process (2/2) (8e2c0bbac797c05df06647f3fab3ba18) switched from
CANCELING to CANCELED.
2020-06-15 02:12:14,551 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom
Source -> Process (1/2) (a01ae675df3bd2537c3882405f7d7a0d) switched from
CANCELING to CANCELED.
2020-06-15 02:12:14,551 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job
tracker-backendflink (fdd30ef6be7b873aabbd4e1c6db23f0b) switched from state
CANCELLING to CANCELED.
2020-06-15 02:12:14,551 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping
checkpoint coordinator for job fdd30ef6be7b873aabbd4e1c6db23f0b.
2020-06-15 02:12:14,551 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
Shutting down
2020-06-15 02:12:14,567 INFO
org.apache.flink.runtime.zookeeper.ZooKeeperStateHandleStore - Removing
/flink-bdp/application_1549925837808_10660736/checkpoints/fdd30ef6be7b873aabbd4e1c6db23f0b
from ZooKeeper
2020-06-15 02:12:14,572 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter - Shutting
down.
2020-06-15 02:12:14,573 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter - Removing
/checkpoint-counter/fdd30ef6be7b873aabbd4e1c6db23f0b from ZooKeeper
2020-06-15 02:12:14,587 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job
fdd30ef6be7b873aabbd4e1c6db23f0b reached globally terminal state CANCELED.
2020-06-15 02:12:14,654 INFO org.apache.flink.runtime.jobmaster.JobMaster -
Stopping the JobMaster for job
tracker-backendflink(fdd30ef6be7b873aabbd4e1c6db23f0b).
2020-06-15 02:12:14,659 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
Stopping ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2020-06-15 02:12:14,662 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Suspending
SlotPool.
2020-06-15 02:12:14,662 INFO org.apache.flink.runtime.jobmaster.JobMaster -
Close ResourceManager connection c56ee819700906d5237f93ee1d68ef06:
JobManager is shutting down..
2020-06-15 02:12:14,662 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Stopping
SlotPool.