flink kafka相关任务执行出错 导致 taskmanager 退出

你好，flink 运行过程中出现以下异常，有何建议指导排除异常，


2019-07-23 10:52:01,420 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.9.0.1
2019-07-23 10:52:01,420 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 23c69d62a0cabf06
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from RUNNING to CANCELING.
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,563 INFO  org.apache.kafka.clients.producer.KafkaProducer               - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2019-07-23 10:52:01,571 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from CANCELING to CANCELED.
2019-07-23 10:52:01,571 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,572 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) [CANCELED]
2019-07-23 10:52:01,572 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed 02dd0b294a75cb672899e83c53985034.
2019-07-23 10:52:01,574 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler       - FATAL: Thread 'flink-metrics-57' produced an uncaught exception. Stopping the process...
java.lang.NoClassDefFoundError: org/apache/kafka/common/metrics/stats/Rate$1
at org.apache.kafka.common.metrics.stats.Rate.convert(Rate.java:98)
at org.apache.kafka.common.metrics.stats.Rate.measure(Rate.java:67)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:61)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:52)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:39)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:29)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:254)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$200(MetricDumpSerialization.java:53)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:166)
at org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:118)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.metrics.stats.Rate$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
2019-07-23 10:52:01,576 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2019-07-23 10:52:01,576 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
2019-07-23 10:52:01,577 INFO  org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager  - Shutting down TaskExecutorLocalStateStoresManager.
2019-07-23 10:52:01,580 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /tmp/flink-io-3e88d484-6424-4ae6-8eae-1694fa1ed526




[hidden email]

看看classpath里有没有kafka-clients jar包,这个类是在这个里面的Rate类没有找到如果有那么看看你使用的kafka版本和kafka-clients是否匹配,我遇到类似的问题由于版本不同导致有的类找不到
                                   


[hidden email]
 
发件人： [hidden email]
发送时间： 2019-07-25 17:26
收件人： user-zh
主题： flink kafka相关任务执行出错 导致 taskmanager 退出
你好，flink 运行过程中出现以下异常，有何建议指导排除异常，
 
 
2019-07-23 10:52:01,420 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.9.0.1
2019-07-23 10:52:01,420 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 23c69d62a0cabf06
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from RUNNING to CANCELING.
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,563 INFO  org.apache.kafka.clients.producer.KafkaProducer               - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2019-07-23 10:52:01,571 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from CANCELING to CANCELED.
2019-07-23 10:52:01,571 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,572 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) [CANCELED]
2019-07-23 10:52:01,572 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed 02dd0b294a75cb672899e83c53985034.
2019-07-23 10:52:01,574 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler       - FATAL: Thread 'flink-metrics-57' produced an uncaught exception. Stopping the process...
java.lang.NoClassDefFoundError: org/apache/kafka/common/metrics/stats/Rate$1
at org.apache.kafka.common.metrics.stats.Rate.convert(Rate.java:98)
at org.apache.kafka.common.metrics.stats.Rate.measure(Rate.java:67)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:61)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:52)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:39)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:29)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:254)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$200(MetricDumpSerialization.java:53)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:166)
at org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:118)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.metrics.stats.Rate$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
2019-07-23 10:52:01,576 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2019-07-23 10:52:01,576 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
2019-07-23 10:52:01,577 INFO  org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager  - Shutting down TaskExecutorLocalStateStoresManager.
2019-07-23 10:52:01,580 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /tmp/flink-io-3e88d484-6424-4ae6-8eae-1694fa1ed526
 
 
 
 
[hidden email]

谢谢，我们这边排查一下



[hidden email]
 
发件人： [hidden email]
发送时间： 2019-07-25 17:30
收件人： user-zh
主题： 回复: flink kafka相关任务执行出错 导致 taskmanager 退出
 
看看classpath里有没有kafka-clients jar包,这个类是在这个里面的Rate类没有找到如果有那么看看你使用的kafka版本和kafka-clients是否匹配,我遇到类似的问题由于版本不同导致有的类找不到
                                   
 
 
[hidden email]
发件人： [hidden email]
发送时间： 2019-07-25 17:26
收件人： user-zh
主题： flink kafka相关任务执行出错 导致 taskmanager 退出
你好，flink 运行过程中出现以下异常，有何建议指导排除异常，
2019-07-23 10:52:01,420 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.9.0.1
2019-07-23 10:52:01,420 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 23c69d62a0cabf06
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from RUNNING to CANCELING.
2019-07-23 10:52:01,560 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,563 INFO  org.apache.kafka.clients.producer.KafkaProducer               - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2019-07-23 10:52:01,571 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) switched from CANCELING to CANCELED.
2019-07-23 10:52:01,571 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034).
2019-07-23 10:52:01,572 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed (3/3) (02dd0b294a75cb672899e83c53985034) [CANCELED]
2019-07-23 10:52:01,572 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source -> Map -> Filter -> Map -> Map -> to: Row -> Map -> Sink: Unnamed 02dd0b294a75cb672899e83c53985034.
2019-07-23 10:52:01,574 ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler       - FATAL: Thread 'flink-metrics-57' produced an uncaught exception. Stopping the process...
java.lang.NoClassDefFoundError: org/apache/kafka/common/metrics/stats/Rate$1
at org.apache.kafka.common.metrics.stats.Rate.convert(Rate.java:98)
at org.apache.kafka.common.metrics.stats.Rate.measure(Rate.java:67)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:61)
at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:52)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:39)
at org.apache.flink.streaming.connectors.kafka.internal.metrics.KafkaMetricMutableWrapper.getValue(KafkaMetricMutableWrapper.java:29)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.serializeGauge(MetricDumpSerialization.java:254)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization.access$200(MetricDumpSerialization.java:53)
at org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricDumpSerializer.serialize(MetricDumpSerialization.java:166)
at org.apache.flink.runtime.metrics.dump.MetricQueryService.onReceive(MetricQueryService.java:118)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.metrics.stats.Rate$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
2019-07-23 10:52:01,576 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2019-07-23 10:52:01,576 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
2019-07-23 10:52:01,577 INFO  org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager  - Shutting down TaskExecutorLocalStateStoresManager.
2019-07-23 10:52:01,580 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /tmp/flink-io-3e88d484-6424-4ae6-8eae-1694fa1ed526
[hidden email]