flink遇到 valueState 自身的 NPE
Locked
 
|
早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的
``` 2020-01-09 05:14:04.087 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) switched from RUNNING to FAILED. java.lang.NullPointerException: null at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) at org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748) 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - Calculating tasks to restart to recover the failed task 90bea66de1c231edf33913ecd54406c1_2. 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 tasks should be restarted to recover the failed task 90bea66de1c231edf33913ecd54406c1_2. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from RUNNING to CANCELING. 2020-01-09 05:14:04.096 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.101 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.103 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.115 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.119 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding the results produced by task execution de624cf5c9d4dec6fe68d4800c701457. 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding the results produced by task execution 60496dddb4bc885ee37a6025662080ad. 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding the results produced by task execution 4a157d98db2f2efad72035af279433ff. 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched from CANCELING to CANCELED. 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - Finally restart 12 tasks to recover from task failure. 2020-01-09 05:14:04.144 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring job 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: Checkpoint 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from CREATED to SCHEDULED. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Filter (2/6) (attempt #2) to container_e06_1577354972821_672000_01_000003 @ rsync.slave713.mars.hadoop.js.ted (dataPort=15754) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Filter (4/6) (attempt #2) to container_e06_1577354972821_672000_01_000002 @ rsync.slave713.mars.hadoop.js.ted (dataPort=20658) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Filter (1/6) (attempt #2) to container_e06_1577354972821_672000_01_000003 @ rsync.slave713.mars.hadoop.js.ted (dataPort=15754) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Filter (3/6) (attempt #2) to container_e06_1577354972821_672000_01_000002 @ rsync.slave713.mars.hadoop.js.ted (dataPort=20658) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Filter (5/6) (attempt #2) to container_e06_1577354972821_672000_01_000003 @ rsync.slave713.mars.hadoop.js.ted (dataPort=15754) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Filter (6/6) (attempt #2) to container_e06_1577354972821_672000_01_000002 @ rsync.slave713.mars.hadoop.js.ted (dataPort=20658) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Filter -> Sink: Unnamed (1/6) (attempt #2) to container_e06_1577354972821_672000_01_000002 @ rsync.slave713.mars.hadoop.js.ted (dataPort=20658) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Filter -> Sink: Unnamed (2/6) (attempt #2) to container_e06_1577354972821_672000_01_000003 @ rsync.slave713.mars.hadoop.js.ted (dataPort=15754) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Filter -> Sink: Unnamed (3/6) (attempt #2) to container_e06_1577354972821_672000_01_000003 @ rsync.slave713.mars.hadoop.js.ted (dataPort=15754) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Filter -> Sink: Unnamed (4/6) (attempt #2) to container_e06_1577354972821_672000_01_000003 @ rsync.slave713.mars.hadoop.js.ted (dataPort=15754) 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Filter -> Sink: Unnamed (6/6) (attempt #2) to container_e06_1577354972821_672000_01_000002 @ rsync.slave713.mars.hadoop.js.ted (dataPort=20658) 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from SCHEDULED to DEPLOYING. 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Filter -> Sink: Unnamed (5/6) (attempt #2) to container_e06_1577354972821_672000_01_000002 @ rsync.slave713.mars.hadoop.js.ted (dataPort=20658) 2020-01-09 05:14:04.159 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.160 [flink-akka.actor.default-dispatcher-28] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.161 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.168 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.169 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.173 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.210 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.230 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.231 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.245 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from DEPLOYING to RUNNING. 2020-01-09 05:14:04.922 [flink-akka.actor.default-dispatcher-14] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from RUNNING to FAILED. java.lang.NullPointerException: null at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) at org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748) ``` 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 ``` if (null != uniqMark.value() && uniqMark.value()) { ``` 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl ``` @Override public void open(Configuration parameters) throws Exception { ValueStateDescriptor<Boolean> descriptor = new ValueStateDescriptor<>("uniqMark", TypeInformation.of(Boolean.class)); StateTtlConfig ttlConfig = StateTtlConfig.newBuilder(Time.hours(1)) .setUpdateType(UpdateType.OnCreateAndWrite) .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) .cleanupIncrementally(1000, false) .build(); descriptor.enableTimeToLive(ttlConfig); uniqMark = getRuntimeContext().getState(descriptor); } ``` 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ 求各位指正 |
|
|
hi Kevin，
能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > ``` > 2020-01-09 05:14:04.087 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) switched from > RUNNING to FAILED. > java.lang.NullPointerException: null > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > at > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > at > org.apache.flink.streaming.runtime.io > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > at > org.apache.flink.streaming.runtime.io > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > at java.lang.Thread.run(Thread.java:748) > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - Calculating > tasks to restart to recover the failed task > 90bea66de1c231edf33913ecd54406c1_2. > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 tasks should > be restarted to recover the failed task 90bea66de1c231edf33913ecd54406c1_2. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from > RUNNING to CANCELING. > 2020-01-09 05:14:04.096 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.101 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.103 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.115 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.119 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding the > results produced by task execution de624cf5c9d4dec6fe68d4800c701457. > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding the > results produced by task execution 60496dddb4bc885ee37a6025662080ad. > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding the > results produced by task execution 4a157d98db2f2efad72035af279433ff. > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched from > CANCELING to CANCELED. > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - Finally > restart 12 tasks to recover from task failure. > 2020-01-09 05:14:04.144 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring job > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: Checkpoint > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master > state to restore > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > CREATED to SCHEDULED. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Filter (2/6) (attempt #2) to > container_e06_1577354972821_672000_01_000003 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Filter (4/6) (attempt #2) to > container_e06_1577354972821_672000_01_000002 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Filter (1/6) (attempt #2) to > container_e06_1577354972821_672000_01_000003 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Filter (3/6) (attempt #2) to > container_e06_1577354972821_672000_01_000002 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Filter (5/6) (attempt #2) to > container_e06_1577354972821_672000_01_000003 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Filter (6/6) (attempt #2) to > container_e06_1577354972821_672000_01_000002 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > container_e06_1577354972821_672000_01_000002 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > container_e06_1577354972821_672000_01_000003 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > container_e06_1577354972821_672000_01_000003 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > container_e06_1577354972821_672000_01_000003 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > container_e06_1577354972821_672000_01_000002 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > SCHEDULED to DEPLOYING. > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > container_e06_1577354972821_672000_01_000002 @ > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > 2020-01-09 05:14:04.159 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.160 [flink-akka.actor.default-dispatcher-28] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.161 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.168 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.169 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.173 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.210 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.230 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.231 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.245 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > DEPLOYING to RUNNING. > 2020-01-09 05:14:04.922 [flink-akka.actor.default-dispatcher-14] INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Filter -> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > RUNNING to FAILED. > java.lang.NullPointerException: null > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > at > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > at > org.apache.flink.streaming.runtime.io > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > at > org.apache.flink.streaming.runtime.io > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > at java.lang.Thread.run(Thread.java:748) > ``` > > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > ``` > > if (null != uniqMark.value() && uniqMark.value()) { > > ``` > > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > ``` > > @Override > public void open(Configuration parameters) throws Exception { > ValueStateDescriptor<Boolean> descriptor = new > ValueStateDescriptor<>("uniqMark", > TypeInformation.of(Boolean.class)); > StateTtlConfig ttlConfig = StateTtlConfig.newBuilder(Time.hours(1)) > .setUpdateType(UpdateType.OnCreateAndWrite) > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > .cleanupIncrementally(1000, false) > .build(); > descriptor.enableTimeToLive(ttlConfig); > uniqMark = getRuntimeContext().getState(descriptor); > } > > ``` > > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > 求各位指正 > -- Benchao Li School of Electronics Engineering and Computer Science, Peking University Tel:+86-15650713730 Email: [hidden email]; [hidden email] |
|
|
Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： hi Kevin， |
|
|
hi Kevin,
邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 或者你可以直接贴文字？ Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > >> hi Kevin， >> >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 >> >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： >> >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 >> > >> > ``` >> > 2020-01-09 05:14:04.087 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) switched from >> > RUNNING to FAILED. >> > java.lang.NullPointerException: null >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) >> > at >> > >> > >> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) >> > at >> > org.apache.flink.streaming.runtime.io >> > >> .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) >> > at >> > org.apache.flink.streaming.runtime.io >> > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) >> > at >> > >> > >> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) >> > at >> > >> > >> org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) >> > at >> > >> > >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >> > at java.lang.Thread.run(Thread.java:748) >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - Calculating >> > tasks to restart to recover the failed task >> > 90bea66de1c231edf33913ecd54406c1_2. >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 tasks >> should >> > be restarted to recover the failed task >> 90bea66de1c231edf33913ecd54406c1_2. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from >> > RUNNING to CANCELING. >> > 2020-01-09 05:14:04.096 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.101 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.103 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.115 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.119 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding >> the >> > results produced by task execution de624cf5c9d4dec6fe68d4800c701457. >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding >> the >> > results produced by task execution 60496dddb4bc885ee37a6025662080ad. >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding >> the >> > results produced by task execution 4a157d98db2f2efad72035af279433ff. >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched from >> > CANCELING to CANCELED. >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO >> > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - Finally >> > restart 12 tasks to recover from task failure. >> > 2020-01-09 05:14:04.144 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring >> job >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: >> Checkpoint >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master >> > state to restore >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from >> > CREATED to SCHEDULED. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> > Source: Custom Source -> Filter (2/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000003 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> > Source: Custom Source -> Filter (4/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000002 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> > Source: Custom Source -> Filter (1/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000003 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> > Source: Custom Source -> Filter (3/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000002 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> > Source: Custom Source -> Filter (5/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000003 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> > Source: Custom Source -> Filter (6/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000002 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000002 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000003 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000003 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000003 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000002 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from >> > SCHEDULED to DEPLOYING. >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to >> > container_e06_1577354972821_672000_01_000002 @ >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) >> > 2020-01-09 05:14:04.159 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.160 [flink-akka.actor.default-dispatcher-28] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.161 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.168 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.169 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.173 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.210 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.230 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.231 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.245 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from >> > DEPLOYING to RUNNING. >> > 2020-01-09 05:14:04.922 [flink-akka.actor.default-dispatcher-14] INFO >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Filter -> >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from >> > RUNNING to FAILED. >> > java.lang.NullPointerException: null >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) >> > at >> > >> > >> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) >> > at >> > org.apache.flink.streaming.runtime.io >> > >> .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) >> > at >> > org.apache.flink.streaming.runtime.io >> > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) >> > at >> > >> > >> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) >> > at >> > >> > >> org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) >> > at >> > >> > >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >> > at java.lang.Thread.run(Thread.java:748) >> > ``` >> > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 >> > ``` >> > >> > if (null != uniqMark.value() && uniqMark.value()) { >> > >> > ``` >> > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl >> > >> > ``` >> > >> > @Override >> > public void open(Configuration parameters) throws Exception { >> > ValueStateDescriptor<Boolean> descriptor = new >> > ValueStateDescriptor<>("uniqMark", >> > TypeInformation.of(Boolean.class)); >> > StateTtlConfig ttlConfig = StateTtlConfig.newBuilder(Time.hours(1)) >> > .setUpdateType(UpdateType.OnCreateAndWrite) >> > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) >> > .cleanupIncrementally(1000, false) >> > .build(); >> > descriptor.enableTimeToLive(ttlConfig); >> > uniqMark = getRuntimeContext().getState(descriptor); >> > } >> > >> > ``` >> > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ >> > >> > 求各位指正 >> > >> >> >> -- >> >> Benchao Li >> School of Electronics Engineering and Computer Science, Peking University >> Tel:+86-15650713730 >> Email: [hidden email]; [hidden email] >> > -- Benchao Li School of Electronics Engineering and Computer Science, Peking University Tel:+86-15650713730 Email: [hidden email]; [hidden email] |
|
|
https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft
谢谢，看看能否看见 Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > hi Kevin, > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > 或者你可以直接贴文字？ > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > >> hi Kevin， > >> > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 > >> > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > >> > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > >> > > >> > ``` > >> > 2020-01-09 05:14:04.087 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) switched from > >> > RUNNING to FAILED. > >> > java.lang.NullPointerException: null > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > >> > at > >> > org.apache.flink.streaming.runtime.io > >> > > >> > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > >> > at > >> > org.apache.flink.streaming.runtime.io > >> > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > >> > at java.lang.Thread.run(Thread.java:748) > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > Calculating > >> > tasks to restart to recover the failed task > >> > 90bea66de1c231edf33913ecd54406c1_2. > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] INFO > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 tasks > >> should > >> > be restarted to recover the failed task > >> 90bea66de1c231edf33913ecd54406c1_2. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched > from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched > from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched > from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched > from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched > from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched > from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from > >> > RUNNING to CANCELING. > >> > 2020-01-09 05:14:04.096 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched > from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.101 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched > from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.103 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched > from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.115 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.119 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding > >> the > >> > results produced by task execution de624cf5c9d4dec6fe68d4800c701457. > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched > from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding > >> the > >> > results produced by task execution 60496dddb4bc885ee37a6025662080ad. > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched > from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Discarding > >> the > >> > results produced by task execution 4a157d98db2f2efad72035af279433ff. > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched > from > >> > CANCELING to CANCELED. > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] INFO > >> > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - Finally > >> > restart 12 tasks to recover from task failure. > >> > 2020-01-09 05:14:04.144 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > Restoring > >> job > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: > >> Checkpoint > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No > master > >> > state to restore > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > >> > CREATED to SCHEDULED. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000003 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000002 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000003 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000002 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000003 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000002 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> Map -> > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000002 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> Map -> > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000003 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> Map -> > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000003 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> Map -> > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000003 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> Map -> > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000002 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > >> > SCHEDULED to DEPLOYING. > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > >> Map -> > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > >> > container_e06_1577354972821_672000_01_000002 @ > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > >> > 2020-01-09 05:14:04.159 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.160 [flink-akka.actor.default-dispatcher-28] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.161 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.168 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.169 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.173 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.210 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.230 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.231 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.245 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > >> > DEPLOYING to RUNNING. > >> > 2020-01-09 05:14:04.922 [flink-akka.actor.default-dispatcher-14] INFO > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Filter -> > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > >> > RUNNING to FAILED. > >> > java.lang.NullPointerException: null > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > >> > at > >> > org.apache.flink.streaming.runtime.io > >> > > >> > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > >> > at > >> > org.apache.flink.streaming.runtime.io > >> > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > >> > at > >> > > >> > > >> > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > >> > at java.lang.Thread.run(Thread.java:748) > >> > ``` > >> > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > >> > ``` > >> > > >> > if (null != uniqMark.value() && uniqMark.value()) { > >> > > >> > ``` > >> > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > >> > > >> > ``` > >> > > >> > @Override > >> > public void open(Configuration parameters) throws Exception { > >> > ValueStateDescriptor<Boolean> descriptor = new > >> > ValueStateDescriptor<>("uniqMark", > >> > TypeInformation.of(Boolean.class)); > >> > StateTtlConfig ttlConfig = StateTtlConfig.newBuilder(Time.hours(1)) > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > >> > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > >> > .cleanupIncrementally(1000, false) > >> > .build(); > >> > descriptor.enableTimeToLive(ttlConfig); > >> > uniqMark = getRuntimeContext().getState(descriptor); > >> > } > >> > > >> > ``` > >> > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > >> > > >> > 求各位指正 > >> > > >> > >> > >> -- > >> > >> Benchao Li > >> School of Electronics Engineering and Computer Science, Peking > University > >> Tel:+86-15650713730 > >> Email: [hidden email]; [hidden email] > >> > > > > -- > > Benchao Li > School of Electronics Engineering and Computer Science, Peking University > Tel:+86-15650713730 > Email: [hidden email]; [hidden email] > |
|
|
我这边点开是 403 Forbidden
Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > 谢谢，看看能否看见 > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > hi Kevin, > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > 或者你可以直接贴文字？ > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > >> hi Kevin， > > >> > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 > > >> > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > >> > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > >> > > > >> > ``` > > >> > 2020-01-09 05:14:04.087 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) switched from > > >> > RUNNING to FAILED. > > >> > java.lang.NullPointerException: null > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > >> > at > > >> > org.apache.flink.streaming.runtime.io > > >> > > > >> > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > >> > at > > >> > org.apache.flink.streaming.runtime.io > > >> > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > >> > at java.lang.Thread.run(Thread.java:748) > > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > > Calculating > > >> > tasks to restart to recover the failed task > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 tasks > > >> should > > >> > be restarted to recover the failed task > > >> 90bea66de1c231edf33913ecd54406c1_2. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched > > from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched > > from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched > > from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched > > from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched > > from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched > > from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from > > >> > RUNNING to CANCELING. > > >> > 2020-01-09 05:14:04.096 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched > > from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.101 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched > > from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.103 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched > > from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.115 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.119 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Discarding > > >> the > > >> > results produced by task execution de624cf5c9d4dec6fe68d4800c701457. > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched > > from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Discarding > > >> the > > >> > results produced by task execution 60496dddb4bc885ee37a6025662080ad. > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched > > from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Discarding > > >> the > > >> > results produced by task execution 4a157d98db2f2efad72035af279433ff. > > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched > > from > > >> > CANCELING to CANCELED. > > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > Finally > > >> > restart 12 tasks to recover from task failure. > > >> > 2020-01-09 05:14:04.144 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > > Restoring > > >> job > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: > > >> Checkpoint > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No > > master > > >> > state to restore > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > > from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > > from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > > from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > > from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > > from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > > from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > > >> > CREATED to SCHEDULED. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > > from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000003 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > > from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000002 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > > from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000003 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > > from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000002 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > > from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000003 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > > from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000002 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> Map -> > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000002 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> Map -> > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000003 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> Map -> > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000003 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> Map -> > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000003 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> Map -> > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000002 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > > >> > SCHEDULED to DEPLOYING. > > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > > >> Map -> > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > >> > container_e06_1577354972821_672000_01_000002 @ > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > >> > 2020-01-09 05:14:04.159 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > > from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.160 [flink-akka.actor.default-dispatcher-28] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > > from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.161 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > > from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.168 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.169 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > > from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > > from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.173 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.210 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > >> Custom > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > > from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.230 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.231 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.245 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched from > > >> > DEPLOYING to RUNNING. > > >> > 2020-01-09 05:14:04.922 [flink-akka.actor.default-dispatcher-14] > INFO > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > >> Filter -> > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched from > > >> > RUNNING to FAILED. > > >> > java.lang.NullPointerException: null > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > >> > at > > >> > org.apache.flink.streaming.runtime.io > > >> > > > >> > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > >> > at > > >> > org.apache.flink.streaming.runtime.io > > >> > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > >> > at java.lang.Thread.run(Thread.java:748) > > >> > ``` > > >> > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > >> > ``` > > >> > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > >> > > > >> > ``` > > >> > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > >> > > > >> > ``` > > >> > > > >> > @Override > > >> > public void open(Configuration parameters) throws Exception { > > >> > ValueStateDescriptor<Boolean> descriptor = new > > >> > ValueStateDescriptor<>("uniqMark", > > >> > TypeInformation.of(Boolean.class)); > > >> > StateTtlConfig ttlConfig = > StateTtlConfig.newBuilder(Time.hours(1)) > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > >> > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > >> > .cleanupIncrementally(1000, false) > > >> > .build(); > > >> > descriptor.enableTimeToLive(ttlConfig); > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > >> > } > > >> > > > >> > ``` > > >> > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > >> > > > >> > 求各位指正 > > >> > > > >> > > >> > > >> -- > > >> > > >> Benchao Li > > >> School of Electronics Engineering and Computer Science, Peking > > University > > >> Tel:+86-15650713730 > > >> Email: [hidden email]; [hidden email] > > >> > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: [hidden email]; [hidden email] > > > -- Benchao Li School of Electronics Engineering and Computer Science, Peking University Tel:+86-15650713730 Email: [hidden email]; [hidden email] |
|
|
https://tva4.sinaimg.cn/large/63137227ly1gaqkn1nlykj20mm0wvgq8.jpg
抱歉，再试试这个 Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:13写道： > 我这边点开是 403 Forbidden > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > > > > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > > > 谢谢，看看能否看见 > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > > > hi Kevin, > > > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > > 或者你可以直接贴文字？ > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > > > >> hi Kevin， > > > >> > > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 > > > >> > > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > > >> > > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > > >> > > > > >> > ``` > > > >> > 2020-01-09 05:14:04.087 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) switched > from > > > >> > RUNNING to FAILED. > > > >> > java.lang.NullPointerException: null > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > >> > at > > > >> > org.apache.flink.streaming.runtime.io > > > >> > > > > >> > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > >> > at > > > >> > org.apache.flink.streaming.runtime.io > > > >> > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > >> > at java.lang.Thread.run(Thread.java:748) > > > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > > > Calculating > > > >> > tasks to restart to recover the failed task > > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 > tasks > > > >> should > > > >> > be restarted to recover the failed task > > > >> 90bea66de1c231edf33913ecd54406c1_2. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched > > > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched > > > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched > > > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched > > > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched > > > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched > > > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched > from > > > >> > RUNNING to CANCELING. > > > >> > 2020-01-09 05:14:04.096 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) switched > > > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.101 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) switched > > > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.103 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) switched > > > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.115 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.119 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Discarding > > > >> the > > > >> > results produced by task execution > de624cf5c9d4dec6fe68d4800c701457. > > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) switched > > > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Discarding > > > >> the > > > >> > results produced by task execution > 60496dddb4bc885ee37a6025662080ad. > > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) switched > > > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Discarding > > > >> the > > > >> > results produced by task execution > 4a157d98db2f2efad72035af279433ff. > > > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) switched > > > from > > > >> > CANCELING to CANCELED. > > > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > > Finally > > > >> > restart 12 tasks to recover from task failure. > > > >> > 2020-01-09 05:14:04.144 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > > > Restoring > > > >> job > > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: > > > >> Checkpoint > > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No > > > master > > > >> > state to restore > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > > > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > > > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > > > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > > > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > > > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > > > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched > from > > > >> > CREATED to SCHEDULED. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > > > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > > > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > > > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > > > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > > > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > > > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> Map -> > > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> Map -> > > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> Map -> > > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> Map -> > > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> Map -> > > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched > from > > > >> > SCHEDULED to DEPLOYING. > > > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > > > >> Map -> > > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > >> > 2020-01-09 05:14:04.159 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) switched > > > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.160 [flink-akka.actor.default-dispatcher-28] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) switched > > > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.161 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) switched > > > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.168 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.169 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) switched > > > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) switched > > > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.173 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.210 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > > > >> Custom > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) switched > > > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.230 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.231 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.245 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched > from > > > >> > DEPLOYING to RUNNING. > > > >> > 2020-01-09 05:14:04.922 [flink-akka.actor.default-dispatcher-14] > > INFO > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > > > >> Filter -> > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > from > > > >> > RUNNING to FAILED. > > > >> > java.lang.NullPointerException: null > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > >> > at > > > >> > org.apache.flink.streaming.runtime.io > > > >> > > > > >> > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > >> > at > > > >> > org.apache.flink.streaming.runtime.io > > > >> > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > >> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > >> > at java.lang.Thread.run(Thread.java:748) > > > >> > ``` > > > >> > > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > > >> > ``` > > > >> > > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > > >> > > > > >> > ``` > > > >> > > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > > >> > > > > >> > ``` > > > >> > > > > >> > @Override > > > >> > public void open(Configuration parameters) throws Exception { > > > >> > ValueStateDescriptor<Boolean> descriptor = new > > > >> > ValueStateDescriptor<>("uniqMark", > > > >> > TypeInformation.of(Boolean.class)); > > > >> > StateTtlConfig ttlConfig = > > StateTtlConfig.newBuilder(Time.hours(1)) > > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > > >> > > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > > >> > .cleanupIncrementally(1000, false) > > > >> > .build(); > > > >> > descriptor.enableTimeToLive(ttlConfig); > > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > > >> > } > > > >> > > > > >> > ``` > > > >> > > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > > >> > > > > >> > 求各位指正 > > > >> > > > > >> > > > >> > > > >> -- > > > >> > > > >> Benchao Li > > > >> School of Electronics Engineering and Computer Science, Peking > > > University > > > >> Tel:+86-15650713730 > > > >> Email: [hidden email]; [hidden email] > > > >> > > > > > > > > > > -- > > > > > > Benchao Li > > > School of Electronics Engineering and Computer Science, Peking > University > > > Tel:+86-15650713730 > > > Email: [hidden email]; [hidden email] > > > > > > > > -- > > Benchao Li > School of Electronics Engineering and Computer Science, Peking University > Tel:+86-15650713730 > Email: [hidden email]; [hidden email] > |
|
|
我感觉这个地方好像没有道理会有`uniqMark`变成`null`,
除非是什么地方反序列化出来`StreamMap`，并且没有调用`StreamMap.open()`. 但是看起来`StreamTask`是可以保证先调用`open`，再调用operator的处理函数的。我也看不出来这个地方有什么问题。 Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:15写道： > https://tva4.sinaimg.cn/large/63137227ly1gaqkn1nlykj20mm0wvgq8.jpg > > 抱歉，再试试这个 > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:13写道： > > > 我这边点开是 403 Forbidden > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > > > > > > > > > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > > > > > 谢谢，看看能否看见 > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > > > > > hi Kevin, > > > > > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > > > 或者你可以直接贴文字？ > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > > > > > >> hi Kevin， > > > > >> > > > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 > > > > >> > > > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > > > >> > > > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > > > >> > > > > > >> > ``` > > > > >> > 2020-01-09 05:14:04.087 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) switched > > from > > > > >> > RUNNING to FAILED. > > > > >> > java.lang.NullPointerException: null > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > >> > at > > > > >> > org.apache.flink.streaming.runtime.io > > > > >> > > > > > >> > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > >> > at > > > > >> > org.apache.flink.streaming.runtime.io > > > > >> > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > >> > at > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > > > > Calculating > > > > >> > tasks to restart to recover the failed task > > > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > > > >> > 2020-01-09 05:14:04.088 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 > > tasks > > > > >> should > > > > >> > be restarted to recover the failed task > > > > >> 90bea66de1c231edf33913ecd54406c1_2. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > switched > > > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > switched > > > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > switched > > > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > switched > > > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.089 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > switched > > > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > switched > > > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.090 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched > > from > > > > >> > RUNNING to CANCELING. > > > > >> > 2020-01-09 05:14:04.096 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > switched > > > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.101 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > switched > > > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.103 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > switched > > > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.115 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) switched > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.119 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) switched > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) switched > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.140 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Discarding > > > > >> the > > > > >> > results produced by task execution > > de624cf5c9d4dec6fe68d4800c701457. > > > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > switched > > > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) switched > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.141 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Discarding > > > > >> the > > > > >> > results produced by task execution > > 60496dddb4bc885ee37a6025662080ad. > > > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > switched > > > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) switched > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.142 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Discarding > > > > >> the > > > > >> > results produced by task execution > > 4a157d98db2f2efad72035af279433ff. > > > > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > switched > > > > from > > > > >> > CANCELING to CANCELED. > > > > >> > 2020-01-09 05:14:04.143 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > > > Finally > > > > >> > restart 12 tasks to recover from task failure. > > > > >> > 2020-01-09 05:14:04.144 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > > > > Restoring > > > > >> job > > > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: > > > > >> Checkpoint > > > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No > > > > master > > > > >> > state to restore > > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > switched > > > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > switched > > > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > switched > > > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.145 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > switched > > > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > switched > > > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > switched > > > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.146 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched > > from > > > > >> > CREATED to SCHEDULED. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > switched > > > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > switched > > > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > switched > > > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > switched > > > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > switched > > > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > switched > > > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> Map -> > > > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> Map -> > > > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> Map -> > > > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> Map -> > > > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.147 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> Map -> > > > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched > > from > > > > >> > SCHEDULED to DEPLOYING. > > > > >> > 2020-01-09 05:14:04.148 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Deploying > > > > >> Map -> > > > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > >> > 2020-01-09 05:14:04.159 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > switched > > > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.160 [flink-akka.actor.default-dispatcher-28] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > switched > > > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.161 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > switched > > > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.168 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) switched > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.169 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > switched > > > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > switched > > > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.170 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) switched > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.173 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.210 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Source: > > > > >> Custom > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > switched > > > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.230 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) switched > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.231 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) switched > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.245 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) switched > > from > > > > >> > DEPLOYING to RUNNING. > > > > >> > 2020-01-09 05:14:04.922 [flink-akka.actor.default-dispatcher-14] > > > INFO > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > -> > > > > >> Filter -> > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) switched > > from > > > > >> > RUNNING to FAILED. > > > > >> > java.lang.NullPointerException: null > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > >> > at > > > > >> > org.apache.flink.streaming.runtime.io > > > > >> > > > > > >> > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > >> > at > > > > >> > org.apache.flink.streaming.runtime.io > > > > >> > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > >> > at > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > >> > at > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > >> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > >> > ``` > > > > >> > > > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > > > >> > ``` > > > > >> > > > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > > > >> > > > > > >> > ``` > > > > >> > > > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > > > >> > > > > > >> > ``` > > > > >> > > > > > >> > @Override > > > > >> > public void open(Configuration parameters) throws Exception { > > > > >> > ValueStateDescriptor<Boolean> descriptor = new > > > > >> > ValueStateDescriptor<>("uniqMark", > > > > >> > TypeInformation.of(Boolean.class)); > > > > >> > StateTtlConfig ttlConfig = > > > StateTtlConfig.newBuilder(Time.hours(1)) > > > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > > > >> > > > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > > > >> > .cleanupIncrementally(1000, false) > > > > >> > .build(); > > > > >> > descriptor.enableTimeToLive(ttlConfig); > > > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > > > >> > } > > > > >> > > > > > >> > ``` > > > > >> > > > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > > > >> > > > > > >> > 求各位指正 > > > > >> > > > > > >> > > > > >> > > > > >> -- > > > > >> > > > > >> Benchao Li > > > > >> School of Electronics Engineering and Computer Science, Peking > > > > University > > > > >> Tel:+86-15650713730 > > > > >> Email: [hidden email]; [hidden email] > > > > >> > > > > > > > > > > > > > -- > > > > > > > > Benchao Li > > > > School of Electronics Engineering and Computer Science, Peking > > University > > > > Tel:+86-15650713730 > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: [hidden email]; [hidden email] > > > -- Benchao Li School of Electronics Engineering and Computer Science, Peking University Tel:+86-15650713730 Email: [hidden email]; [hidden email] |
|
|
谢答
但还有一问题想请教： 当 valueState 触发 ttl 被回收后，这里的引用应该也会被 gc 掉，就会变成 null 了吧？难道是 operator 原来处理这个 key 的线程会被一并回收掉，下次这个 key 再来时其实已经是另外新生成的线程提供服务了（这样肯定要重新调用一次 open 方法）？看了代码但还没搞明白，求解惑，谢谢 Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:59写道： > 我感觉这个地方好像没有道理会有`uniqMark`变成`null`, > 除非是什么地方反序列化出来`StreamMap`，并且没有调用`StreamMap.open()`. > 但是看起来`StreamTask`是可以保证先调用`open`，再调用operator的处理函数的。我也看不出来这个地方有什么问题。 > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:15写道： > > > https://tva4.sinaimg.cn/large/63137227ly1gaqkn1nlykj20mm0wvgq8.jpg > > > > 抱歉，再试试这个 > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:13写道： > > > > > 我这边点开是 403 Forbidden > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > > > > > > > > > > > > > > > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > > > > > > > 谢谢，看看能否看见 > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > > > > > > > hi Kevin, > > > > > > > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > > > > 或者你可以直接贴文字？ > > > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > > > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > > > > > > > >> hi Kevin， > > > > > >> > > > > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 > > > > > >> > > > > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > > > > >> > > > > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > > > > >> > > > > > > >> > ``` > > > > > >> > 2020-01-09 05:14:04.087 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) > switched > > > from > > > > > >> > RUNNING to FAILED. > > > > > >> > java.lang.NullPointerException: null > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > >> > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > >> > at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > >> > 2020-01-09 05:14:04.088 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > > > > > Calculating > > > > > >> > tasks to restart to recover the failed task > > > > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > > > > >> > 2020-01-09 05:14:04.088 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 > > > tasks > > > > > >> should > > > > > >> > be restarted to recover the failed task > > > > > >> 90bea66de1c231edf33913ecd54406c1_2. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.090 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.090 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.090 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.096 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.101 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.103 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.115 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.119 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.140 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.140 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Discarding > > > > > >> the > > > > > >> > results produced by task execution > > > de624cf5c9d4dec6fe68d4800c701457. > > > > > >> > 2020-01-09 05:14:04.141 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.141 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.141 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Discarding > > > > > >> the > > > > > >> > results produced by task execution > > > 60496dddb4bc885ee37a6025662080ad. > > > > > >> > 2020-01-09 05:14:04.142 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.142 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.142 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Discarding > > > > > >> the > > > > > >> > results produced by task execution > > > 4a157d98db2f2efad72035af279433ff. > > > > > >> > 2020-01-09 05:14:04.143 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.143 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > > > > Finally > > > > > >> > restart 12 tasks to recover from task failure. > > > > > >> > 2020-01-09 05:14:04.144 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > > > > > Restoring > > > > > >> job > > > > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: > > > > > >> Checkpoint > > > > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > No > > > > > master > > > > > >> > state to restore > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.148 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.148 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.159 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.160 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.161 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.168 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.169 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.170 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.170 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.173 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.210 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.230 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.231 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.245 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.922 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > RUNNING to FAILED. > > > > > >> > java.lang.NullPointerException: null > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > >> > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > >> > at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > >> > ``` > > > > > >> > > > > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > > > > >> > ``` > > > > > >> > > > > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > > > > >> > > > > > > >> > ``` > > > > > >> > > > > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > > > > >> > > > > > > >> > ``` > > > > > >> > > > > > > >> > @Override > > > > > >> > public void open(Configuration parameters) throws Exception { > > > > > >> > ValueStateDescriptor<Boolean> descriptor = new > > > > > >> > ValueStateDescriptor<>("uniqMark", > > > > > >> > TypeInformation.of(Boolean.class)); > > > > > >> > StateTtlConfig ttlConfig = > > > > StateTtlConfig.newBuilder(Time.hours(1)) > > > > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > > > > >> > > > > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > > > > >> > .cleanupIncrementally(1000, false) > > > > > >> > .build(); > > > > > >> > descriptor.enableTimeToLive(ttlConfig); > > > > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > > > > >> > } > > > > > >> > > > > > > >> > ``` > > > > > >> > > > > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > > > > >> > > > > > > >> > 求各位指正 > > > > > >> > > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> > > > > > >> Benchao Li > > > > > >> School of Electronics Engineering and Computer Science, Peking > > > > > University > > > > > >> Tel:+86-15650713730 > > > > > >> Email: [hidden email]; [hidden email] > > > > > >> > > > > > > > > > > > > > > > > -- > > > > > > > > > > Benchao Li > > > > > School of Electronics Engineering and Computer Science, Peking > > > University > > > > > Tel:+86-15650713730 > > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > > > > > > -- > > > > > > Benchao Li > > > School of Electronics Engineering and Computer Science, Peking > University > > > Tel:+86-15650713730 > > > Email: [hidden email]; [hidden email] > > > > > > > > -- > > Benchao Li > School of Electronics Engineering and Computer Science, Peking University > Tel:+86-15650713730 > Email: [hidden email]; [hidden email] > |
|
Hi Kevin
State TTL 是清理的state中的数据条目entry，不是清理state在map函数中的对象本身。所以无论如何，作为value state对象的uniqMark 是不会因为TTL而变成null的。 我注意到你的作业即使发生failover之后，立刻恢复的时候，仍然遇到了这个NPE问题，我怀疑你实际运行的代码第39行并不是你贴出来的代码，很有可能是对应你的代码的第34行，也就是map方法的输入RDLog是null，这也符合作业成功restore之后，又再次立即遇到failover的场景，也就是处理到了非法“脏数据”，导致作业不断failover。建议你按照这个思路确认排除一下。 祝好 唐云 ________________________________ From: Kevin Liao <[hidden email]> Sent: Thursday, January 9, 2020 23:17 To: [hidden email] <[hidden email]> Subject: Re: flink遇到 valueState 自身的 NPE 谢答 但还有一问题想请教： 当 valueState 触发 ttl 被回收后，这里的引用应该也会被 gc 掉，就会变成 null 了吧？难道是 operator 原来处理这个 key 的线程会被一并回收掉，下次这个 key 再来时其实已经是另外新生成的线程提供服务了（这样肯定要重新调用一次 open 方法）？看了代码但还没搞明白，求解惑，谢谢 Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:59写道： > 我感觉这个地方好像没有道理会有`uniqMark`变成`null`, > 除非是什么地方反序列化出来`StreamMap`，并且没有调用`StreamMap.open()`. > 但是看起来`StreamTask`是可以保证先调用`open`，再调用operator的处理函数的。我也看不出来这个地方有什么问题。 > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:15写道： > > > https://tva4.sinaimg.cn/large/63137227ly1gaqkn1nlykj20mm0wvgq8.jpg > > > > 抱歉，再试试这个 > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:13写道： > > > > > 我这边点开是 403 Forbidden > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > > > > > > > > > > > > > > > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > > > > > > > 谢谢，看看能否看见 > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > > > > > > > hi Kevin, > > > > > > > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > > > > 或者你可以直接贴文字？ > > > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > > > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > > > > > > > >> hi Kevin， > > > > > >> > > > > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ 从上面的日志看不出来是valueState是null呢。 > > > > > >> > > > > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > > > > >> > > > > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > > > > >> > > > > > > >> > ``` > > > > > >> > 2020-01-09 05:14:04.087 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) > switched > > > from > > > > > >> > RUNNING to FAILED. > > > > > >> > java.lang.NullPointerException: null > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > >> > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > >> > at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > >> > 2020-01-09 05:14:04.088 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > > > > > Calculating > > > > > >> > tasks to restart to recover the failed task > > > > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > > > > >> > 2020-01-09 05:14:04.088 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - 12 > > > tasks > > > > > >> should > > > > > >> > be restarted to recover the failed task > > > > > >> 90bea66de1c231edf33913ecd54406c1_2. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.089 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.090 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.090 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > switched > > > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.090 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > switched > > > from > > > > > >> > RUNNING to CANCELING. > > > > > >> > 2020-01-09 05:14:04.096 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.101 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.103 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.115 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.119 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.140 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.140 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Discarding > > > > > >> the > > > > > >> > results produced by task execution > > > de624cf5c9d4dec6fe68d4800c701457. > > > > > >> > 2020-01-09 05:14:04.141 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.141 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.141 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Discarding > > > > > >> the > > > > > >> > results produced by task execution > > > 60496dddb4bc885ee37a6025662080ad. > > > > > >> > 2020-01-09 05:14:04.142 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.142 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > switched > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.142 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Discarding > > > > > >> the > > > > > >> > results produced by task execution > > > 4a157d98db2f2efad72035af279433ff. > > > > > >> > 2020-01-09 05:14:04.143 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > switched > > > > > from > > > > > >> > CANCELING to CANCELED. > > > > > >> > 2020-01-09 05:14:04.143 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > > > > Finally > > > > > >> > restart 12 tasks to recover from task failure. > > > > > >> > 2020-01-09 05:14:04.144 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > > > > > Restoring > > > > > >> job > > > > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid checkpoint: > > > > > >> Checkpoint > > > > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > No > > > > > master > > > > > >> > state to restore > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.145 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > switched > > > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.146 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > switched > > > from > > > > > >> > CREATED to SCHEDULED. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > switched > > > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.147 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.148 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > switched > > > from > > > > > >> > SCHEDULED to DEPLOYING. > > > > > >> > 2020-01-09 05:14:04.148 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Deploying > > > > > >> Map -> > > > > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > >> > 2020-01-09 05:14:04.159 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.160 > [flink-akka.actor.default-dispatcher-28] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.161 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.168 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.169 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.170 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.170 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.173 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.210 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Source: > > > > > >> Custom > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > switched > > > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.230 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.231 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.245 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > switched > > > from > > > > > >> > DEPLOYING to RUNNING. > > > > > >> > 2020-01-09 05:14:04.922 > [flink-akka.actor.default-dispatcher-14] > > > > INFO > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map > > -> > > > > > >> Filter -> > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > switched > > > from > > > > > >> > RUNNING to FAILED. > > > > > >> > java.lang.NullPointerException: null > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > >> > at com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > >> > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > >> > at > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > >> > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > >> > at > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > >> > at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > >> > ``` > > > > > >> > > > > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > > > > >> > ``` > > > > > >> > > > > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > > > > >> > > > > > > >> > ``` > > > > > >> > > > > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > > > > >> > > > > > > >> > ``` > > > > > >> > > > > > > >> > @Override > > > > > >> > public void open(Configuration parameters) throws Exception { > > > > > >> > ValueStateDescriptor<Boolean> descriptor = new > > > > > >> > ValueStateDescriptor<>("uniqMark", > > > > > >> > TypeInformation.of(Boolean.class)); > > > > > >> > StateTtlConfig ttlConfig = > > > > StateTtlConfig.newBuilder(Time.hours(1)) > > > > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > > > > >> > > > > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > > > > >> > .cleanupIncrementally(1000, false) > > > > > >> > .build(); > > > > > >> > descriptor.enableTimeToLive(ttlConfig); > > > > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > > > > >> > } > > > > > >> > > > > > > >> > ``` > > > > > >> > > > > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > > > > >> > > > > > > >> > 求各位指正 > > > > > >> > > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> > > > > > >> Benchao Li > > > > > >> School of Electronics Engineering and Computer Science, Peking > > > > > University > > > > > >> Tel:+86-15650713730 > > > > > >> Email: [hidden email]; [hidden email] > > > > > >> > > > > > > > > > > > > > > > > -- > > > > > > > > > > Benchao Li > > > > > School of Electronics Engineering and Computer Science, Peking > > > University > > > > > Tel:+86-15650713730 > > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > > > > > > -- > > > > > > Benchao Li > > > School of Electronics Engineering and Computer Science, Peking > University > > > Tel:+86-15650713730 > > > Email: [hidden email]; [hidden email] > > > > > > > > -- > > Benchao Li > School of Electronics Engineering and Computer Science, Peking University > Tel:+86-15650713730 > Email: [hidden email]; [hidden email] > |
|
|
谢答，首先贴的代码确实是运行的程序
此外刚刚又通过打印 log 确认了 uniqMark == null 是 false 我现在的怀疑点是这个地方 if (null != uniqMark.value() && uniqMark.value()) { 第一处uniqMark.value()可以取到结果（不为 null），同时由于 ttl 策略会触发 clear，进而导致第二个uniqMark.value()取出来的就是 null 了（看来编译器没有对这种写法做优化，就是真实的执行了两次value()函数），追了下代码似乎可以支持我的猜测，所以准备明天验证下这个猜测 明天有消息同步，谢谢 Yun Tang <[hidden email]> 于2020年1月10日周五 上午12:59写道： > Hi Kevin > > State TTL 是清理的state中的数据条目entry，不是清理state在map函数中的对象本身。所以无论如何，作为value > state对象的uniqMark 是不会因为TTL而变成null的。 > > > 我注意到你的作业即使发生failover之后，立刻恢复的时候，仍然遇到了这个NPE问题，我怀疑你实际运行的代码第39行并不是你贴出来的代码，很有可能是对应你的代码的第34行，也就是map方法的输入RDLog是null，这也符合作业成功restore之后，又再次立即遇到failover的场景，也就是处理到了非法“脏数据”，导致作业不断failover。建议你按照这个思路确认排除一下。 > > 祝好 > 唐云 > ------------------------------ > *From:* Kevin Liao <[hidden email]> > *Sent:* Thursday, January 9, 2020 23:17 > *To:* [hidden email] <[hidden email]> > *Subject:* Re: flink遇到 valueState 自身的 NPE > > 谢答 > 但还有一问题想请教： > > 当 valueState 触发 ttl 被回收后，这里的引用应该也会被 gc 掉，就会变成 null 了吧？难道是 operator 原来处理这个 > key 的线程会被一并回收掉，下次这个 key 再来时其实已经是另外新生成的线程提供服务了（这样肯定要重新调用一次 open > 方法）？看了代码但还没搞明白，求解惑，谢谢 > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:59写道： > > > 我感觉这个地方好像没有道理会有`uniqMark`变成`null`, > > 除非是什么地方反序列化出来`StreamMap`，并且没有调用`StreamMap.open()`. > > 但是看起来`StreamTask`是可以保证先调用`open`，再调用operator的处理函数的。我也看不出来这个地方有什么问题。 > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:15写道： > > > > > https://tva4.sinaimg.cn/large/63137227ly1gaqkn1nlykj20mm0wvgq8.jpg > > > > > > 抱歉，再试试这个 > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:13写道： > > > > > > > 我这边点开是 403 Forbidden > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > > > > > > > > > > > > > > > > > > > > > > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > > > > > > > > > 谢谢，看看能否看见 > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > > > > > > > > > hi Kevin, > > > > > > > > > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > > > > > 或者你可以直接贴文字？ > > > > > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > > > > > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > > > > > > > > > >> hi Kevin， > > > > > > >> > > > > > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ > 从上面的日志看不出来是valueState是null呢。 > > > > > > >> > > > > > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > > > > > >> > > > > > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > 2020-01-09 05:14:04.087 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) > > switched > > > > from > > > > > > >> > RUNNING to FAILED. > > > > > > >> > java.lang.NullPointerException: null > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > >> > > > > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > > >> > at > > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > > >> > 2020-01-09 05:14:04.088 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > > > > > > Calculating > > > > > > >> > tasks to restart to recover the failed task > > > > > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > > > > > >> > 2020-01-09 05:14:04.088 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > 12 > > > > tasks > > > > > > >> should > > > > > > >> > be restarted to recover the failed task > > > > > > >> 90bea66de1c231edf33913ecd54406c1_2. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.090 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.090 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.090 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.096 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.101 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.103 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.115 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.119 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.140 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.140 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Discarding > > > > > > >> the > > > > > > >> > results produced by task execution > > > > de624cf5c9d4dec6fe68d4800c701457. > > > > > > >> > 2020-01-09 05:14:04.141 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.141 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.141 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Discarding > > > > > > >> the > > > > > > >> > results produced by task execution > > > > 60496dddb4bc885ee37a6025662080ad. > > > > > > >> > 2020-01-09 05:14:04.142 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.142 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.142 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Discarding > > > > > > >> the > > > > > > >> > results produced by task execution > > > > 4a157d98db2f2efad72035af279433ff. > > > > > > >> > 2020-01-09 05:14:04.143 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.143 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > > > > > Finally > > > > > > >> > restart 12 tasks to recover from task failure. > > > > > > >> > 2020-01-09 05:14:04.144 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator > - > > > > > > Restoring > > > > > > >> job > > > > > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid > checkpoint: > > > > > > >> Checkpoint > > > > > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator > - > > No > > > > > > master > > > > > > >> > state to restore > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.148 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.148 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.159 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.160 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.161 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.168 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.169 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.170 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.170 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.173 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.210 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.230 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.231 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.245 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.922 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > RUNNING to FAILED. > > > > > > >> > java.lang.NullPointerException: null > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > >> > > > > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > > >> > at > > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > @Override > > > > > > >> > public void open(Configuration parameters) throws Exception > { > > > > > > >> > ValueStateDescriptor<Boolean> descriptor = new > > > > > > >> > ValueStateDescriptor<>("uniqMark", > > > > > > >> > TypeInformation.of(Boolean.class)); > > > > > > >> > StateTtlConfig ttlConfig = > > > > > StateTtlConfig.newBuilder(Time.hours(1)) > > > > > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > > > > > >> > > > > > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > > > > > >> > .cleanupIncrementally(1000, false) > > > > > > >> > .build(); > > > > > > >> > descriptor.enableTimeToLive(ttlConfig); > > > > > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > > > > > >> > } > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > > > > > >> > > > > > > > >> > 求各位指正 > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> -- > > > > > > >> > > > > > > >> Benchao Li > > > > > > >> School of Electronics Engineering and Computer Science, Peking > > > > > > University > > > > > > >> Tel:+86-15650713730 > > > > > > >> Email: [hidden email]; [hidden email] > > > > > > >> > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Benchao Li > > > > > > School of Electronics Engineering and Computer Science, Peking > > > > University > > > > > > Tel:+86-15650713730 > > > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Benchao Li > > > > School of Electronics Engineering and Computer Science, Peking > > University > > > > Tel:+86-15650713730 > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: [hidden email]; [hidden email] > > > |
|
|
Hi Kevin
帮你看了下代码，问题定位到了。 容易发生的原因在于你的可见性配置成了 ReturnExpiredIfNotCleanedUp，而Flink会先发现数据expire，然后触发clear操作，之后再返回expire的数据 [1]。建议修改一下第39行的判断条件，每次取数据时都做是否为null的判断。 [1] https://github.com/apache/flink/blob/dba5b9e0138b667c3ecd32f7b16645d531477720/flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/AbstractTtlDecorator.java#L96 祝好 唐云 ________________________________ From: Kevin Liao <[hidden email]> Sent: Friday, January 10, 2020 1:08 To: Yun Tang <[hidden email]> Cc: [hidden email] <[hidden email]> Subject: Re: flink遇到 valueState 自身的 NPE 谢答，首先贴的代码确实是运行的程序 此外刚刚又通过打印 log 确认了 uniqMark == null 是 false 我现在的怀疑点是这个地方 if (null != uniqMark.value() && uniqMark.value()) { 第一处uniqMark.value()可以取到结果（不为 null），同时由于 ttl 策略会触发 clear，进而导致第二个uniqMark.value()取出来的就是 null 了（看来编译器没有对这种写法做优化，就是真实的执行了两次value()函数），追了下代码似乎可以支持我的猜测，所以准备明天验证下这个猜测 明天有消息同步，谢谢 Yun Tang <[hidden email]> 于2020年1月10日周五 上午12:59写道： > Hi Kevin > > State TTL 是清理的state中的数据条目entry，不是清理state在map函数中的对象本身。所以无论如何，作为value > state对象的uniqMark 是不会因为TTL而变成null的。 > > > 我注意到你的作业即使发生failover之后，立刻恢复的时候，仍然遇到了这个NPE问题，我怀疑你实际运行的代码第39行并不是你贴出来的代码，很有可能是对应你的代码的第34行，也就是map方法的输入RDLog是null，这也符合作业成功restore之后，又再次立即遇到failover的场景，也就是处理到了非法“脏数据”，导致作业不断failover。建议你按照这个思路确认排除一下。 > > 祝好 > 唐云 > ------------------------------ > *From:* Kevin Liao <[hidden email]> > *Sent:* Thursday, January 9, 2020 23:17 > *To:* [hidden email] <[hidden email]> > *Subject:* Re: flink遇到 valueState 自身的 NPE > > 谢答 > 但还有一问题想请教： > > 当 valueState 触发 ttl 被回收后，这里的引用应该也会被 gc 掉，就会变成 null 了吧？难道是 operator 原来处理这个 > key 的线程会被一并回收掉，下次这个 key 再来时其实已经是另外新生成的线程提供服务了（这样肯定要重新调用一次 open > 方法）？看了代码但还没搞明白，求解惑，谢谢 > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:59写道： > > > 我感觉这个地方好像没有道理会有`uniqMark`变成`null`, > > 除非是什么地方反序列化出来`StreamMap`，并且没有调用`StreamMap.open()`. > > 但是看起来`StreamTask`是可以保证先调用`open`，再调用operator的处理函数的。我也看不出来这个地方有什么问题。 > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:15写道： > > > > > https://tva4.sinaimg.cn/large/63137227ly1gaqkn1nlykj20mm0wvgq8.jpg > > > > > > 抱歉，再试试这个 > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:13写道： > > > > > > > 我这边点开是 403 Forbidden > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > > > > > > > > > > > > > > > > > > > > > > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > > > > > > > > > 谢谢，看看能否看见 > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > > > > > > > > > hi Kevin, > > > > > > > > > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > > > > > 或者你可以直接贴文字？ > > > > > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > > > > > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > > > > > > > > > >> hi Kevin， > > > > > > >> > > > > > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ > 从上面的日志看不出来是valueState是null呢。 > > > > > > >> > > > > > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > > > > > >> > > > > > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > 2020-01-09 05:14:04.087 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) > > switched > > > > from > > > > > > >> > RUNNING to FAILED. > > > > > > >> > java.lang.NullPointerException: null > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > >> > > > > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > > >> > at > > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > > >> > 2020-01-09 05:14:04.088 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > > > > > > Calculating > > > > > > >> > tasks to restart to recover the failed task > > > > > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > > > > > >> > 2020-01-09 05:14:04.088 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy - > 12 > > > > tasks > > > > > > >> should > > > > > > >> > be restarted to recover the failed task > > > > > > >> 90bea66de1c231edf33913ecd54406c1_2. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.089 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.090 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.090 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > > switched > > > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.090 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > > switched > > > > from > > > > > > >> > RUNNING to CANCELING. > > > > > > >> > 2020-01-09 05:14:04.096 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.101 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.103 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.115 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.119 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.140 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.140 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Discarding > > > > > > >> the > > > > > > >> > results produced by task execution > > > > de624cf5c9d4dec6fe68d4800c701457. > > > > > > >> > 2020-01-09 05:14:04.141 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.141 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.141 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Discarding > > > > > > >> the > > > > > > >> > results produced by task execution > > > > 60496dddb4bc885ee37a6025662080ad. > > > > > > >> > 2020-01-09 05:14:04.142 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.142 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > > switched > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.142 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Discarding > > > > > > >> the > > > > > > >> > results produced by task execution > > > > 4a157d98db2f2efad72035af279433ff. > > > > > > >> > 2020-01-09 05:14:04.143 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > > switched > > > > > > from > > > > > > >> > CANCELING to CANCELED. > > > > > > >> > 2020-01-09 05:14:04.143 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > > > > > Finally > > > > > > >> > restart 12 tasks to recover from task failure. > > > > > > >> > 2020-01-09 05:14:04.144 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator > - > > > > > > Restoring > > > > > > >> job > > > > > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid > checkpoint: > > > > > > >> Checkpoint > > > > > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator > - > > No > > > > > > master > > > > > > >> > state to restore > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.145 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > switched > > > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.146 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > switched > > > > from > > > > > > >> > CREATED to SCHEDULED. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > switched > > > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.147 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.148 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > switched > > > > from > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > >> > 2020-01-09 05:14:04.148 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Deploying > > > > > > >> Map -> > > > > > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > >> > 2020-01-09 05:14:04.159 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.160 > > [flink-akka.actor.default-dispatcher-28] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.161 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.168 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.169 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.170 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.170 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.173 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.210 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > Source: > > > > > > >> Custom > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > switched > > > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.230 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.231 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.245 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > switched > > > > from > > > > > > >> > DEPLOYING to RUNNING. > > > > > > >> > 2020-01-09 05:14:04.922 > > [flink-akka.actor.default-dispatcher-14] > > > > > INFO > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > Map > > > -> > > > > > > >> Filter -> > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > switched > > > > from > > > > > > >> > RUNNING to FAILED. > > > > > > >> > java.lang.NullPointerException: null > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > > >> > at > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > >> > > > > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > > >> > at > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > >> > > > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > > >> > at > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > > >> > at > > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > > >> > at > > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > @Override > > > > > > >> > public void open(Configuration parameters) throws Exception > { > > > > > > >> > ValueStateDescriptor<Boolean> descriptor = new > > > > > > >> > ValueStateDescriptor<>("uniqMark", > > > > > > >> > TypeInformation.of(Boolean.class)); > > > > > > >> > StateTtlConfig ttlConfig = > > > > > StateTtlConfig.newBuilder(Time.hours(1)) > > > > > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > > > > > >> > > > > > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > > > > > >> > .cleanupIncrementally(1000, false) > > > > > > >> > .build(); > > > > > > >> > descriptor.enableTimeToLive(ttlConfig); > > > > > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > > > > > >> > } > > > > > > >> > > > > > > > >> > ``` > > > > > > >> > > > > > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > > > > > >> > > > > > > > >> > 求各位指正 > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> -- > > > > > > >> > > > > > > >> Benchao Li > > > > > > >> School of Electronics Engineering and Computer Science, Peking > > > > > > University > > > > > > >> Tel:+86-15650713730 > > > > > > >> Email: [hidden email]; [hidden email] > > > > > > >> > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Benchao Li > > > > > > School of Electronics Engineering and Computer Science, Peking > > > > University > > > > > > Tel:+86-15650713730 > > > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Benchao Li > > > > School of Electronics Engineering and Computer Science, Peking > > University > > > > Tel:+86-15650713730 > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: [hidden email]; [hidden email] > > > |
|
|
谢谢，同定位到这个地方
已改成 ``` Boolean existed = uniqMark.value(); // 如果是重复出现则被过滤掉 if (null != existed && existed) { return null; } uniqMark.update(true); ``` 问题解决 Yun Tang <[hidden email]> 于2020年1月10日周五 上午3:05写道： > Hi Kevin > > 帮你看了下代码，问题定位到了。 > 容易发生的原因在于你的可见性配置成了 ReturnExpiredIfNotCleanedUp，而Flink会先发现数据expire，然后触发clear操作，之后再返回expire的数据 > [1]。建议修改一下第39行的判断条件，每次取数据时都做是否为null的判断。 > > > [1] > https://github.com/apache/flink/blob/dba5b9e0138b667c3ecd32f7b16645d531477720/flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/AbstractTtlDecorator.java#L96 > > 祝好 > 唐云 > > ------------------------------ > *From:* Kevin Liao <[hidden email]> > *Sent:* Friday, January 10, 2020 1:08 > *To:* Yun Tang <[hidden email]> > *Cc:* [hidden email] <[hidden email]> > *Subject:* Re: flink遇到 valueState 自身的 NPE > > 谢答，首先贴的代码确实是运行的程序 > > 此外刚刚又通过打印 log 确认了 uniqMark == null 是 false > > 我现在的怀疑点是这个地方 > > if (null != uniqMark.value() && uniqMark.value()) { > > 第一处uniqMark.value()可以取到结果（不为 null），同时由于 ttl 策略会触发 > clear，进而导致第二个uniqMark.value()取出来的就是 null > 了（看来编译器没有对这种写法做优化，就是真实的执行了两次value()函数），追了下代码似乎可以支持我的猜测，所以准备明天验证下这个猜测 > > 明天有消息同步，谢谢 > > Yun Tang <[hidden email]> 于2020年1月10日周五 上午12:59写道： > > > Hi Kevin > > > > State TTL 是清理的state中的数据条目entry，不是清理state在map函数中的对象本身。所以无论如何，作为value > > state对象的uniqMark 是不会因为TTL而变成null的。 > > > > > > > 我注意到你的作业即使发生failover之后，立刻恢复的时候，仍然遇到了这个NPE问题，我怀疑你实际运行的代码第39行并不是你贴出来的代码，很有可能是对应你的代码的第34行，也就是map方法的输入RDLog是null，这也符合作业成功restore之后，又再次立即遇到failover的场景，也就是处理到了非法“脏数据”，导致作业不断failover。建议你按照这个思路确认排除一下。 > > > > 祝好 > > 唐云 > > ------------------------------ > > *From:* Kevin Liao <[hidden email]> > > *Sent:* Thursday, January 9, 2020 23:17 > > *To:* [hidden email] <[hidden email]> > > *Subject:* Re: flink遇到 valueState 自身的 NPE > > > > 谢答 > > 但还有一问题想请教： > > > > 当 valueState 触发 ttl 被回收后，这里的引用应该也会被 gc 掉，就会变成 null 了吧？难道是 operator 原来处理这个 > > key 的线程会被一并回收掉，下次这个 key 再来时其实已经是另外新生成的线程提供服务了（这样肯定要重新调用一次 open > > 方法）？看了代码但还没搞明白，求解惑，谢谢 > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:59写道： > > > > > 我感觉这个地方好像没有道理会有`uniqMark`变成`null`, > > > 除非是什么地方反序列化出来`StreamMap`，并且没有调用`StreamMap.open()`. > > > 但是看起来`StreamTask`是可以保证先调用`open`，再调用operator的处理函数的。我也看不出来这个地方有什么问题。 > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:15写道： > > > > > > > https://tva4.sinaimg.cn/large/63137227ly1gaqkn1nlykj20mm0wvgq8.jpg > > > > > > > > 抱歉，再试试这个 > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:13写道： > > > > > > > > > 我这边点开是 403 Forbidden > > > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午8:09写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://gm1.ggpht.com/FZGtbLggyPPZ_BoU0gt2SQTv7fyhNOKu03ZjsKq7G6DtqWJ5DY0NmL-2s64P-LUzedbTm8DE_FeggNtPAb4VEmypAoPfW8VFSFxOWxMGBvMi5G6xHoZ3THKPYHnAj8KydQ02OjvV-R3IEwBvwIDnZRmwCv3ohyjPF76gbvOOYrKzgaLb_pykWsQDpvROHr3lgU2rezH33Jt3xJEOjXGjHsUFUxiil0PYkQFdA0BP77lypYQLw4RL8BxMz3HfaCiNAGb_q5w8JNmckHLU3g9EuPgtqj6WP3XDv07PBuCXMvmfNcFbAciMeJuOOeE8VBqDCacjuiDtJzVrK1boxcBnzFvT_QazOwaJ27SSuJ_u5KCerTURen2vLBF1RN-x9eOVz9wg6w1oXyMAF7LMjGsYsVzUu3It5AyzLkm-_znosNtAJp2AW_qGmGo-k02fcrMjUoELiGvqn6W1kScnFI4gNWi_dpZe0Uoq1zF2m1crww1oNGOeRjFlCK_-iC19CPfsTVCtwN3tdKnaKdLe2TbfVdFA0DnBUz8NrhV-mvmZlEwi9-ngK-WOy8yjA4fin1zaE2SJCf2zfBSZwGR2eY_E_WZQiFRmSBI2A7vpoyFvTV3E99MIi0MC5PUAeRiu4v4JIVDkV_yUGIUvoa7pxdf7LpZN_DbikQVk7yES8kxxL5qG2Eae8vftWJuBVi5mWTxvElBgInyUntobXHdxfb2YR4JdBgVPN5QionJiIc9g5i0ClGECZbyHPbsQy4pEVw=s0-l75-ft-l75-ft > > > > > > > > > > > > 谢谢，看看能否看见 > > > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午8:07写道： > > > > > > > > > > > > > hi Kevin, > > > > > > > > > > > > > > 邮件里面贴不了图片，如果要贴图片，需要用一些第三方的图床工具。 > > > > > > > 或者你可以直接贴文字？ > > > > > > > > > > > > > > Kevin Liao <[hidden email]> 于2020年1月9日周四 下午7:10写道： > > > > > > > > > > > > > > > [image: B40C260D-DCC3-4B7D-A024-3839803C2234.png] > > > > > > > > > > > > > > > > Benchao Li <[hidden email]> 于2020年1月9日周四 下午6:42写道： > > > > > > > > > > > > > > > >> hi Kevin， > > > > > > > >> > > > > > > > >> 能贴一下MyMapFunction2.java:39 这里的代码吗？ > > 从上面的日志看不出来是valueState是null呢。 > > > > > > > >> > > > > > > > >> Kevin Liao <[hidden email]> 于2020年1月9日周四 下午5:57写道： > > > > > > > >> > > > > > > > >> > 早上发现任务异常，task 在不停重启，遂查看 jm 日志，最开始的报错是这样的 > > > > > > > >> > > > > > > > > >> > ``` > > > > > > > >> > 2020-01-09 05:14:04.087 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (3/6) (d0e6c4a05d0274c18a4a3df41ab5ff1b) > > > switched > > > > > from > > > > > > > >> > RUNNING to FAILED. > > > > > > > >> > java.lang.NullPointerException: null > > > > > > > >> > at > > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > > > >> > at > > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > > > >> > at > > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > > > >> > at > > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > > >> > > > > > > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > > > >> > at > > > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > > > >> > at > > > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > > > >> > 2020-01-09 05:14:04.088 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy > - > > > > > > > Calculating > > > > > > > >> > tasks to restart to recover the failed task > > > > > > > >> > 90bea66de1c231edf33913ecd54406c1_2. > > > > > > > >> > 2020-01-09 05:14:04.088 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > o.a.f.r.e.failover.flip1.RestartPipelinedRegionStrategy > - > > 12 > > > > > tasks > > > > > > > >> should > > > > > > > >> > be restarted to recover the failed task > > > > > > > >> 90bea66de1c231edf33913ecd54406c1_2. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > > > switched > > > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > > > switched > > > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > > > switched > > > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > > > switched > > > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > > > switched > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > > > switched > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > > > switched > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.089 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > > > switched > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.090 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > > > switched > > > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.090 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > > > switched > > > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.090 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > > > switched > > > > > from > > > > > > > >> > RUNNING to CANCELING. > > > > > > > >> > 2020-01-09 05:14:04.096 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (1/6) (4c7cd6eaf5c3ca9c2b0db73e7d230a9e) > > > > switched > > > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.101 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (2/6) (bb14d5776c53babcc57edd65bf7159b0) > > > > switched > > > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.103 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (5/6) (cf4ff0c189315b27e7e2178d8c60e49f) > > > > switched > > > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.115 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (2/6) (bf8515b4f9e852182a5519102fe4fdf3) > > > switched > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.119 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (4/6) (e52c1e70884a6599205f9e0f5b092bc0) > > > switched > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.140 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (6/6) (de624cf5c9d4dec6fe68d4800c701457) > > > switched > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.140 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > > Discarding > > > > > > > >> the > > > > > > > >> > results produced by task execution > > > > > de624cf5c9d4dec6fe68d4800c701457. > > > > > > > >> > 2020-01-09 05:14:04.141 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (6/6) (ac52050e60236cd1efcd262c8240cd25) > > > > switched > > > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.141 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (5/6) (60496dddb4bc885ee37a6025662080ad) > > > switched > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.141 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > > Discarding > > > > > > > >> the > > > > > > > >> > results produced by task execution > > > > > 60496dddb4bc885ee37a6025662080ad. > > > > > > > >> > 2020-01-09 05:14:04.142 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (4/6) (8c8b07cb39a3f682f41f102e614765e6) > > > > switched > > > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.142 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (1/6) (4a157d98db2f2efad72035af279433ff) > > > switched > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.142 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > > Discarding > > > > > > > >> the > > > > > > > >> > results produced by task execution > > > > > 4a157d98db2f2efad72035af279433ff. > > > > > > > >> > 2020-01-09 05:14:04.143 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (3/6) (34aadddbffe9f61b1916bcd1427ced96) > > > > switched > > > > > > > from > > > > > > > >> > CANCELING to CANCELED. > > > > > > > >> > 2020-01-09 05:14:04.143 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > > > o.a.f.r.e.failover.AdaptedRestartPipelinedRegionStrategyNG - > > > > > > Finally > > > > > > > >> > restart 12 tasks to recover from task failure. > > > > > > > >> > 2020-01-09 05:14:04.144 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator > > - > > > > > > > Restoring > > > > > > > >> job > > > > > > > >> > 7907d93e17fccf460069c2ca7f13d7e7 from latest valid > > checkpoint: > > > > > > > >> Checkpoint > > > > > > > >> > 720 @ 1578518011978 for 7907d93e17fccf460069c2ca7f13d7e7. > > > > > > > >> > 2020-01-09 05:14:04.145 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.checkpoint.CheckpointCoordinator > > - > > > No > > > > > > > master > > > > > > > >> > state to restore > > > > > > > >> > 2020-01-09 05:14:04.145 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > > switched > > > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.145 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > > switched > > > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.145 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > > switched > > > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.145 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > > switched > > > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.146 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > > switched > > > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.146 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > > switched > > > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.146 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > > switched > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.146 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > > switched > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.146 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > > switched > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.146 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > > switched > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.146 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > > switched > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > > switched > > > > > from > > > > > > > >> > CREATED to SCHEDULED. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > > switched > > > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> > Source: Custom Source -> Filter (2/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > > switched > > > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> > Source: Custom Source -> Filter (4/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > > switched > > > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> > Source: Custom Source -> Filter (1/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > > switched > > > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> > Source: Custom Source -> Filter (3/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > > switched > > > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> > Source: Custom Source -> Filter (5/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > > switched > > > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> > Source: Custom Source -> Filter (6/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > > switched > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> Map -> > > > > > > > >> > Filter -> Sink: Unnamed (1/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > > switched > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> Map -> > > > > > > > >> > Filter -> Sink: Unnamed (2/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > > switched > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> Map -> > > > > > > > >> > Filter -> Sink: Unnamed (3/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > > switched > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> Map -> > > > > > > > >> > Filter -> Sink: Unnamed (4/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000003 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=15754) > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > > switched > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.147 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> Map -> > > > > > > > >> > Filter -> Sink: Unnamed (6/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > > >> > 2020-01-09 05:14:04.148 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > > switched > > > > > from > > > > > > > >> > SCHEDULED to DEPLOYING. > > > > > > > >> > 2020-01-09 05:14:04.148 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > > Deploying > > > > > > > >> Map -> > > > > > > > >> > Filter -> Sink: Unnamed (5/6) (attempt #2) to > > > > > > > >> > container_e06_1577354972821_672000_01_000002 @ > > > > > > > >> > rsync.slave713.mars.hadoop.js.ted (dataPort=20658) > > > > > > > >> > 2020-01-09 05:14:04.159 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (1/6) (ffa3b56b4e1321aa5ec94e040921e223) > > > > switched > > > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.160 > > > [flink-akka.actor.default-dispatcher-28] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (2/6) (d83eee0b90a0e4a4dcff3e1ca6bed8b6) > > > > switched > > > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.161 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (5/6) (7a2252b0a18cd91211adae3ce3b613be) > > > > switched > > > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.168 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (4/6) (9ad2c9ee62ff804683284b0cdf571c99) > > > switched > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.169 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (4/6) (da6e91a7475712da5ca52fc82069345f) > > > > switched > > > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.170 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (6/6) (6010486f42fc1e0e7c93623174bf0477) > > > > switched > > > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.170 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (2/6) (af4e3fc75729a2d035db42ec95e44189) > > > switched > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.173 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > > switched > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.210 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > > > Source: > > > > > > > >> Custom > > > > > > > >> > Source -> Filter (3/6) (5c3fb959f79d53cdca506188ed40cc6d) > > > > switched > > > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.230 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (1/6) (eb06b4212d17f10cf4a8bcc812eaa073) > > > switched > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.231 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (6/6) (1e66dc5791940cf7e213b1b3cf8cf1fd) > > > switched > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.245 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (5/6) (4cb656b991141acbbe9e1033103ab470) > > > switched > > > > > from > > > > > > > >> > DEPLOYING to RUNNING. > > > > > > > >> > 2020-01-09 05:14:04.922 > > > [flink-akka.actor.default-dispatcher-14] > > > > > > INFO > > > > > > > >> > org.apache.flink.runtime.executiongraph.ExecutionGraph - > > Map > > > > -> > > > > > > > >> Filter -> > > > > > > > >> > Sink: Unnamed (3/6) (adb9b0aabe0234fe336211f5e44e18f3) > > > switched > > > > > from > > > > > > > >> > RUNNING to FAILED. > > > > > > > >> > java.lang.NullPointerException: null > > > > > > > >> > at > > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:39) > > > > > > > >> > at > > com.sogou.qidian.MyMapFunction2.map(MyMapFunction2.java:25) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > > > > > >> > at > > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > .StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) > > > > > > > >> > at > > > > > > > >> > org.apache.flink.streaming.runtime.io > > > > > > > >> > > > > > > > > > > > > > .StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301) > > > > > > > >> > at > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406) > > > > > > > >> > at > > > > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) > > > > > > > >> > at > > > org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) > > > > > > > >> > at java.lang.Thread.run(Thread.java:748) > > > > > > > >> > ``` > > > > > > > >> > > > > > > > > >> > 之后就是不停的循环，怪异的是 MyMapFunction2.java:39内容是 > > > > > > > >> > ``` > > > > > > > >> > > > > > > > > >> > if (null != uniqMark.value() && uniqMark.value()) { > > > > > > > >> > > > > > > > > >> > ``` > > > > > > > >> > > > > > > > > >> > 而报错的时间点距离任务启动差不多就是 12 小时，符合我设置的 valueState 的 ttl > > > > > > > >> > > > > > > > > >> > ``` > > > > > > > >> > > > > > > > > >> > @Override > > > > > > > >> > public void open(Configuration parameters) throws > Exception > > { > > > > > > > >> > ValueStateDescriptor<Boolean> descriptor = new > > > > > > > >> > ValueStateDescriptor<>("uniqMark", > > > > > > > >> > TypeInformation.of(Boolean.class)); > > > > > > > >> > StateTtlConfig ttlConfig = > > > > > > StateTtlConfig.newBuilder(Time.hours(1)) > > > > > > > >> > .setUpdateType(UpdateType.OnCreateAndWrite) > > > > > > > >> > > > > > > > .setStateVisibility(StateVisibility.ReturnExpiredIfNotCleanedUp) > > > > > > > >> > .cleanupIncrementally(1000, false) > > > > > > > >> > .build(); > > > > > > > >> > descriptor.enableTimeToLive(ttlConfig); > > > > > > > >> > uniqMark = getRuntimeContext().getState(descriptor); > > > > > > > >> > } > > > > > > > >> > > > > > > > > >> > ``` > > > > > > > >> > > > > > > > > >> > 所以我猜测在什么极端情况下会出现 valueState 自身变为 null 的情况么？（比如刚刚被清理的瞬间）？ > > > > > > > >> > > > > > > > > >> > 求各位指正 > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> -- > > > > > > > >> > > > > > > > >> Benchao Li > > > > > > > >> School of Electronics Engineering and Computer Science, > Peking > > > > > > > University > > > > > > > >> Tel:+86-15650713730 > > > > > > > >> Email: [hidden email]; [hidden email] > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Benchao Li > > > > > > > School of Electronics Engineering and Computer Science, Peking > > > > > University > > > > > > > Tel:+86-15650713730 > > > > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Benchao Li > > > > > School of Electronics Engineering and Computer Science, Peking > > > University > > > > > Tel:+86-15650713730 > > > > > Email: [hidden email]; [hidden email] > > > > > > > > > > > > > > > > > > -- > > > > > > Benchao Li > > > School of Electronics Engineering and Computer Science, Peking > University > > > Tel:+86-15650713730 > > > Email: [hidden email]; [hidden email] > > > > > > |
«
Return to Apache Flink 中文用户邮件列表
|
1 view|%1 views
| Free forum by Nabble | Edit this page |