Apache Flink 中文用户邮件列表

flink kafka相关任务执行出错导致 taskmanager 退出

Classic

List

Threaded

4 messages Options

rockeycui@163.com

flink kafka相关任务执行出错导致 taskmanager 退出

你好，flink 运行过程中出现以下异常，有何建议指导排除异常，

2019-07-23 10:52:01,420 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.9.0.1
2019-07-23 10:52:01,420 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 23c69d62a0cabf06
2019-07-23 10:52:01,560 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to cancel task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,560 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from RUNNING to CANCELING.
2019-07-23 10:52:01,560 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,563 INFO org.apache.kafka.clients.producer.KafkaProducer - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2019-07-23 10:52:01,571 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from CANCELING to CANCELED.
2019-07-23 10:52:01,571 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,572 INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) [CANCELED]
2019-07-23 10:52:01,572 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed 02dd0b294a75cb672899e83c53985034.
2019-07-23 10:52:01,574 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread 'flink-metrics-57' produced an uncaught exception. Stopping the process...
java.lang.NoClassDefFoundError: org/apache/kafka/common/metrics/stats/Rate$1
at org.apache.kafka.common.metrics.stats.Rate.convert(Rate.java:98)
at org.apache.kafka.common.metrics.stats.Rate.measure(Rate.java:67)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:61)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:52)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:39)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:29)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:254)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$200(MetricDumpSerialization.java:53)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:166)
at org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:118)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.metrics.stats.Rate$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
2019-07-23 10:52:01,576 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
2019-07-23 10:52:01,576 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Shutting down BLOB cache
2019-07-23 10:52:01,577 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager - Shutting down TaskExecutorLocalStateStoresManager.
2019-07-23 10:52:01,580 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager removed spill file directory /tmp/flink-io-3e88d484-6424-4ae6-8eae-1694fa1ed526

[hidden email]

zhisheng

Re: flink kafka相关任务执行出错导致 taskmanager 退出

hi，rockeycui
你是使用了 flink-metrics-prometheus 去 report metrics 数据吗？看到不少和 metrics 相关的字眼

[hidden email] <[hidden email]> 于2019年7月25日周四下午5:12写道：

> 你好，flink 运行过程中出现以下异常，有何建议指导排除异常，
>
> 2019-07-23 10:52:01,420 INFO org.apache.kafka.common.utils.AppInfoParser
> - Kafka version : 0.9.0.1
> 2019-07-23 10:52:01,420 INFO org.apache.kafka.common.utils.AppInfoParser
> - Kafka commitId : 23c69d62a0cabf06
> 2019-07-23 10:52:01,560 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to cancel task Source: Custom Source -> Map
> -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3)
> (02dd0b294a75cb672899e83c53985034).
> 2019-07-23 10:52:01,560 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source -> Map -> Filter -> Map -> Map ->
> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034)
> switched from RUNNING to CANCELING.
> 2019-07-23 10:52:01,560 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Source: Custom
> Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed
> (3/3) (02dd0b294a75cb672899e83c53985034).
> 2019-07-23 10:52:01,563 INFO
> org.apache.kafka.clients.producer.KafkaProducer - Closing the
> Kafka producer with timeoutMillis = 9223372036854775807 ms.
> 2019-07-23 10:52:01,571 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source -> Map -> Filter -> Map -> Map ->
> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034)
> switched from CANCELING to CANCELED.
> 2019-07-23 10:52:01,571 INFO org.apache.flink.runtime.taskmanager.Task
> - Freeing task resources for Source: Custom Source -> Map
> -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3)
> (02dd0b294a75cb672899e83c53985034).
> 2019-07-23 10:52:01,572 INFO org.apache.flink.runtime.taskmanager.Task
> - Ensuring all FileSystem streams are closed for task
> Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map ->
> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) [CANCELED]
> 2019-07-23 10:52:01,572 INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor -
> Un-registering task and sending final execution state CANCELED to
> JobManager for task Source: Custom Source -> Map -> Filter -> Map -> Map ->
> to: Row -> Map -> Sink: Unnamed 02dd0b294a75cb672899e83c53985034.
> 2019-07-23 10:52:01,574 ERROR
> org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL:
> Thread 'flink-metrics-57' produced an uncaught exception. Stopping the
> process...
> java.lang.NoClassDefFoundError:
> org/apache/kafka/common/metrics/stats/Rate$1
> at org.apache.kafka.common.metrics.stats.Rate.convert(Rate.java:98)
> at org.apache.kafka.common.metrics.stats.Rate.measure(Rate.java:67)
> at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:61)
> at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:52)
> at
> org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:39)
> at
> org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:29)
> at
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:254)
> at
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$200(MetricDumpSerialization.java:53)
> at
> org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:166)
> at
> org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:118)
> at
> akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.kafka.common.metrics.stats.Rate$1
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 20 more
> 2019-07-23 10:52:01,576 INFO
> org.apache.flink.runtime.blob.TransientBlobCache - Shutting
> down BLOB cache
> 2019-07-23 10:52:01,576 INFO
> org.apache.flink.runtime.blob.PermanentBlobCache - Shutting
> down BLOB cache
> 2019-07-23 10:52:01,577 INFO
> org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager -
> Shutting down TaskExecutorLocalStateStoresManager.
> 2019-07-23 10:52:01,580 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager
> - I/O manager removed spill file directory
> /tmp/flink-io-3e88d484-6424-4ae6-8eae-1694fa1ed526
>
>
>
>
> [hidden email]
>

rockeycui@163.com

Re: Re: flink kafka相关任务执行出错导致 taskmanager 退出

你好，zhisheng，
我们并没有用到 flink-metrics-prometheus，只是任务执行的时候就会报出找不到 org/apache/kafka/common/metrics/stats/Rate$1 这个类，不知道是否是flink自己调用到的

[hidden email]

发件人： zhisheng
发送时间： 2019-07-25 17:27
收件人： user-zh
主题： Re: flink kafka相关任务执行出错导致 taskmanager 退出
hi，rockeycui
你是使用了 flink-metrics-prometheus 去 report metrics 数据吗？看到不少和 metrics 相关的字眼

[hidden email] <[hidden email]> 于2019年7月25日周四下午5:12写道：

九思

回复： Re: flink kafka相关任务执行出错导致 taskmanager 退出

会不会包冲突了。看看你的jar里面是不是有一些flink的包，把它去掉。集群会有flink相关包的

------------------ 原始邮件 ------------------
发件人: "[hidden email]"<[hidden email]>;
发送时间: 2019年7月26日(星期五) 上午10:22
收件人: "user-zh"<[hidden email]>;

主题: Re: Re: flink kafka相关任务执行出错导致 taskmanager 退出

你好，zhisheng，
我们并没有用到 flink-metrics-prometheus，只是任务执行的时候就会报出找不到 org/apache/kafka/common/metrics/stats/Rate$1 这个类，不知道是否是flink自己调用到的

[hidden email]

发件人： zhisheng
发送时间： 2019-07-25 17:27
收件人： user-zh
主题： Re: flink kafka相关任务执行出错导致 taskmanager 退出
hi，rockeycui
你是使用了 flink-metrics-prometheus 去 report metrics 数据吗？看到不少和 metrics 相关的字眼

[hidden email] <[hidden email]> 于2019年7月25日周四下午5:12写道：

flink kafka相关任务执行出错 导致 taskmanager 退出

flink kafka相关任务执行出错 导致 taskmanager 退出

Re: flink kafka相关任务执行出错 导致 taskmanager 退出

Re: Re: flink kafka相关任务执行出错 导致 taskmanager 退出

回复： Re: flink kafka相关任务执行出错 导致 taskmanager 退出

flink kafka相关任务执行出错导致 taskmanager 退出

flink kafka相关任务执行出错导致 taskmanager 退出

Re: flink kafka相关任务执行出错导致 taskmanager 退出

Re: Re: flink kafka相关任务执行出错导致 taskmanager 退出

回复： Re: flink kafka相关任务执行出错导致 taskmanager 退出