rocksdb作为状态后端任务重启时，恢复作业失败Could not restore keyed state backend for KeyedProcessOperator

[chenxyz]

任务启用rocksdb作为状态后端，任务出现异常重启时经常失败Could not restore keyed state backend for KeyedProcessOperator。这个问题怎么解决呢？

版本：1.10 standalone

配置信息：

state.backend: rocksdb

state.checkpoints.dir: hdfs://nameservice1/data/flink1_10/checkpoint

state.backend.incremental: true

jobmanager.execution.failover-strategy: region

io.tmp.dirs: /data/flink1_10/tmp




任务的checkpoint配置：

env.enableCheckpointing(2 * 60 * 1000);

env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);

env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2 * 60 * 1000);

env.getCheckpointConfig().setCheckpointTimeout(60000);

env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

env.getCheckpointConfig().setTolerableCheckpointFailureNumber(12);

env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);




日志信息：




2020-04-01 11:13:03

java.lang.Exception: Exception while creating StreamOperatorStateContext.

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:191)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)

at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)

at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)

at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)

at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)

at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926_(1/2) from any of the 1 provided restore options.

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)

... 9 more

Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected exception.

at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)

at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

... 11 more

Caused by: java.nio.file.NoSuchFileException: /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst -> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/3e979cc5-82c1-42bf-a269-1ce6e43f3e10/001888.sst

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)

at java.nio.file.Files.createLink(Files.java:1086)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:480)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)

at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)

... 15 more







TaskManager的报错信息：




2020-04-01 14:48:10,726 ERROR org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder  - Caught unexpected exception.

java.io.InterruptedIOException: Interrupted while waiting for data to be acknowledged by pipeline

at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)

at org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)

at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)

at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)

at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)

at org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)

at org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)

at org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)

at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)

at org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)

at java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)

at java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)

at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)

at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)

at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)

at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)

at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)

at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)

at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)

at java.lang.Thread.run(Thread.java:748)

2020-04-01 14:48:10,726 WARN  org.apache.flink.streaming.api.operators.BackendRestorerProcedure  - Exception while restoring keyed state backend for KeyedProcessOperator_0cead2d40df6e304d3168f6366c79e3f_(1/2) from alternative (1/1), will retry while more alternatives are available.

org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected exception.

at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)

at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)

at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)

at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)

at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)

at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)

at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.InterruptedIOException: Interrupted while waiting for data to be acknowledged by pipeline

at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)

at org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)

at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)

at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)

at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)

at org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)

at org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)

at org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)

at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)

at org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)

at java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)

at java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)

at org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)

at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)

at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)

... 15 more

[Congxian Qiu]

Hi
Restore 可以大致分为两部分，1）下载文件；2）从下载的文件恢复
从 TM 日志看像下载出错了，你可以看下
/data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
这个文件是不是存在 double check 下，如果是下载失败，你需要确认下下载失败的原因

Best,
Congxian


chenxyz <[hidden email]> 于2020年4月1日周三 下午3:02写道：

> 任务启用rocksdb作为状态后端，任务出现异常重启时经常失败Could not restore keyed state backend for
> KeyedProcessOperator。这个问题怎么解决呢？
>
> 版本：1.10 standalone
>
> 配置信息：
>
> state.backend: rocksdb
>
> state.checkpoints.dir: hdfs://nameservice1/data/flink1_10/checkpoint
>
> state.backend.incremental: true
>
> jobmanager.execution.failover-strategy: region
>
> io.tmp.dirs: /data/flink1_10/tmp
>
>
>
>
> 任务的checkpoint配置：
>
> env.enableCheckpointing(2 * 60 * 1000);
>
>
> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>
> env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2 * 60 * 1000);
>
> env.getCheckpointConfig().setCheckpointTimeout(60000);
>
> env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
>
> env.getCheckpointConfig().setTolerableCheckpointFailureNumber(12);
>
>
> env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
>
>
>
>
> 日志信息：
>
>
>
>
> 2020-04-01 11:13:03
>
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:191)
>
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for
> KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926_(1/2) from any of the
> 1 provided restore options.
>
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>
> ... 9 more
>
> Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught
> unexpected exception.
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>
> ... 11 more
>
> Caused by: java.nio.file.NoSuchFileException:
> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
> ->
> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/3e979cc5-82c1-42bf-a269-1ce6e43f3e10/001888.sst
>
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
>
> at java.nio.file.Files.createLink(Files.java:1086)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:480)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>
> ... 15 more
>
>
>
>
>
>
>
> TaskManager的报错信息：
>
>
>
>
> 2020-04-01 14:48:10,726 ERROR
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder  -
> Caught unexpected exception.
>
> java.io.InterruptedIOException: Interrupted while waiting for data to be
> acknowledged by pipeline
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
>
> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
>
> at
> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
>
> at
> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
>
> at
> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
>
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
>
> at
> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
>
> at
> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-01 14:48:10,726 WARN
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure  -
> Exception while restoring keyed state backend for
> KeyedProcessOperator_0cead2d40df6e304d3168f6366c79e3f_(1/2) from
> alternative (1/1), will retry while more alternatives are available.
>
> org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected
> exception.
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.io.InterruptedIOException: Interrupted while waiting for
> data to be acknowledged by pipeline
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
>
> at
> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
>
> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
>
> at
> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
>
> at
> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
>
> at
> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
>
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
>
> at
> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
>
> at
> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>
> ... 15 more

[chenxyz]

Hi, 从贤，
我查看了下HDFS, /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160下面是空的，也没有db这一层目录。








在 2020-04-01 16:50:13，"Congxian Qiu" <[hidden email]> 写道：

>Hi
>Restore 可以大致分为两部分，1）下载文件；2）从下载的文件恢复
>从 TM 日志看像下载出错了，你可以看下
>/data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
>这个文件是不是存在 double check 下，如果是下载失败，你需要确认下下载失败的原因
>
>Best,
>Congxian
>
>
>chenxyz <[hidden email]> 于2020年4月1日周三 下午3:02写道：
>
>> 任务启用rocksdb作为状态后端，任务出现异常重启时经常失败Could not restore keyed state backend for
>> KeyedProcessOperator。这个问题怎么解决呢？
>>
>> 版本：1.10 standalone
>>
>> 配置信息：
>>
>> state.backend: rocksdb
>>
>> state.checkpoints.dir: hdfs://nameservice1/data/flink1_10/checkpoint
>>
>> state.backend.incremental: true
>>
>> jobmanager.execution.failover-strategy: region
>>
>> io.tmp.dirs: /data/flink1_10/tmp
>>
>>
>>
>>
>> 任务的checkpoint配置：
>>
>> env.enableCheckpointing(2 * 60 * 1000);
>>
>>
>> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>>
>> env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2 * 60 * 1000);
>>
>> env.getCheckpointConfig().setCheckpointTimeout(60000);
>>
>> env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
>>
>> env.getCheckpointConfig().setTolerableCheckpointFailureNumber(12);
>>
>>
>> env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
>>
>>
>>
>>
>> 日志信息：
>>
>>
>>
>>
>> 2020-04-01 11:13:03
>>
>> java.lang.Exception: Exception while creating StreamOperatorStateContext.
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:191)
>>
>> at
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>>
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>>
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
>> state backend for
>> KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926_(1/2) from any of the
>> 1 provided restore options.
>>
>> at
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>>
>> ... 9 more
>>
>> Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught
>> unexpected exception.
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>>
>> at
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>>
>> at
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>>
>> ... 11 more
>>
>> Caused by: java.nio.file.NoSuchFileException:
>> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
>> ->
>> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/3e979cc5-82c1-42bf-a269-1ce6e43f3e10/001888.sst
>>
>> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>>
>> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>>
>> at
>> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
>>
>> at java.nio.file.Files.createLink(Files.java:1086)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:480)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>>
>> ... 15 more
>>
>>
>>
>>
>>
>>
>>
>> TaskManager的报错信息：
>>
>>
>>
>>
>> 2020-04-01 14:48:10,726 ERROR
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder  -
>> Caught unexpected exception.
>>
>> java.io.InterruptedIOException: Interrupted while waiting for data to be
>> acknowledged by pipeline
>>
>> at
>> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
>>
>> at
>> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
>>
>> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
>>
>> at
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>>
>> at
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
>>
>> at
>> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
>>
>> at
>> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
>>
>> at
>> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
>>
>> at
>> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>>
>> at
>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
>>
>> at
>> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
>>
>> at
>> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>>
>> at
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>>
>> at
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>>
>> at
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>>
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>>
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> 2020-04-01 14:48:10,726 WARN
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure  -
>> Exception while restoring keyed state backend for
>> KeyedProcessOperator_0cead2d40df6e304d3168f6366c79e3f_(1/2) from
>> alternative (1/1), will retry while more alternatives are available.
>>
>> org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected
>> exception.
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>>
>> at
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>>
>> at
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>>
>> at
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>>
>> at
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>>
>> at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>>
>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>>
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Caused by: java.io.InterruptedIOException: Interrupted while waiting for
>> data to be acknowledged by pipeline
>>
>> at
>> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
>>
>> at
>> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
>>
>> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
>>
>> at
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>>
>> at
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
>>
>> at
>> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
>>
>> at
>> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
>>
>> at
>> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
>>
>> at
>> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>>
>> at
>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
>>
>> at
>> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
>>
>> at
>> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>>
>> at
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>>
>> at
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>>
>> ... 15 more

[Congxian Qiu]

HDFS 上的路径和本地不一样，如果你要看 HDFS 路径的话，可能需要看 Checkjpoint Meta 的相关信息，这个比较麻烦，可以参考
CheckpointMetadataLoadingTest 的相关测试。
我再看了一下你给的 TM Log，看上去是 148 行的 outputStream.close() 出错了（有个比较奇怪的现象是，这里的
outputStream 是本地的文件，但是从错误栈看是 HadoopFileSystem）。你这个是稳定复现的问题吗？如果是的话，能否贴一下打开
debug log，贴一下 JM/TM log，另外能给一个可复现的 作业更好

Best,
Congxian


chenxyz <[hidden email]> 于2020年4月1日周三 下午5:18写道：

> Hi, 从贤，
> 我查看了下HDFS,
> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160下面是空的，也没有db这一层目录。
>
>
>
>
>
>
>
>
> 在 2020-04-01 16:50:13，"Congxian Qiu" <[hidden email]> 写道：
> >Hi
> >Restore 可以大致分为两部分，1）下载文件；2）从下载的文件恢复
> >从 TM 日志看像下载出错了，你可以看下
>
> >/data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
> >这个文件是不是存在 double check 下，如果是下载失败，你需要确认下下载失败的原因
> >
> >Best,
> >Congxian
> >
> >
> >chenxyz <[hidden email]> 于2020年4月1日周三 下午3:02写道：
> >
> >> 任务启用rocksdb作为状态后端，任务出现异常重启时经常失败Could not restore keyed state backend for
> >> KeyedProcessOperator。这个问题怎么解决呢？
> >>
> >> 版本：1.10 standalone
> >>
> >> 配置信息：
> >>
> >> state.backend: rocksdb
> >>
> >> state.checkpoints.dir: hdfs://nameservice1/data/flink1_10/checkpoint
> >>
> >> state.backend.incremental: true
> >>
> >> jobmanager.execution.failover-strategy: region
> >>
> >> io.tmp.dirs: /data/flink1_10/tmp
> >>
> >>
> >>
> >>
> >> 任务的checkpoint配置：
> >>
> >> env.enableCheckpointing(2 * 60 * 1000);
> >>
> >>
> >>
> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
> >>
> >> env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2 * 60 * 1000);
> >>
> >> env.getCheckpointConfig().setCheckpointTimeout(60000);
> >>
> >> env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
> >>
> >> env.getCheckpointConfig().setTolerableCheckpointFailureNumber(12);
> >>
> >>
> >>
> env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
> >>
> >>
> >>
> >>
> >> 日志信息：
> >>
> >>
> >>
> >>
> >> 2020-04-01 11:13:03
> >>
> >> java.lang.Exception: Exception while creating
> StreamOperatorStateContext.
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:191)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> >>
> >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
> >>
> >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
> >>
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> >> state backend for
> >> KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926_(1/2) from any of
> the
> >> 1 provided restore options.
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
> >>
> >> ... 9 more
> >>
> >> Caused by: org.apache.flink.runtime.state.BackendBuildingException:
> Caught
> >> unexpected exception.
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> >>
> >> ... 11 more
> >>
> >> Caused by: java.nio.file.NoSuchFileException:
> >>
> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
> >> ->
> >>
> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/3e979cc5-82c1-42bf-a269-1ce6e43f3e10/001888.sst
> >>
> >> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> >>
> >> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> >>
> >> at
> >>
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
> >>
> >> at java.nio.file.Files.createLink(Files.java:1086)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:480)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
> >>
> >> ... 15 more
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> TaskManager的报错信息：
> >>
> >>
> >>
> >>
> >> 2020-04-01 14:48:10,726 ERROR
> >>
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder  -
> >> Caught unexpected exception.
> >>
> >> java.io.InterruptedIOException: Interrupted while waiting for data to
> be
> >> acknowledged by pipeline
> >>
> >> at
> >>
> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
> >>
> >> at
> >>
> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
> >>
> >> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
> >>
> >> at
> >>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> >>
> >> at
> >>
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
> >>
> >> at
> >>
> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
> >>
> >> at
> >>
> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
> >>
> >> at
> >>
> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
> >>
> >> at
> >>
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> >>
> >> at
> >>
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
> >>
> >> at
> >>
> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
> >>
> >> at
> >>
> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> >>
> >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
> >>
> >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
> >>
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> 2020-04-01 14:48:10,726 WARN
> >> org.apache.flink.streaming.api.operators.BackendRestorerProcedure  -
> >> Exception while restoring keyed state backend for
> >> KeyedProcessOperator_0cead2d40df6e304d3168f6366c79e3f_(1/2) from
> >> alternative (1/1), will retry while more alternatives are available.
> >>
> >> org.apache.flink.runtime.state.BackendBuildingException: Caught
> unexpected
> >> exception.
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
> >>
> >> at
> >>
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> >>
> >> at
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> >>
> >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
> >>
> >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
> >>
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> Caused by: java.io.InterruptedIOException: Interrupted while waiting
> for
> >> data to be acknowledged by pipeline
> >>
> >> at
> >>
> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
> >>
> >> at
> >>
> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
> >>
> >> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
> >>
> >> at
> >>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> >>
> >> at
> >>
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
> >>
> >> at
> >>
> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
> >>
> >> at
> >>
> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
> >>
> >> at
> >>
> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
> >>
> >> at
> >>
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> >>
> >> at
> >>
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
> >>
> >> at
> >>
> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
> >>
> >> at
> >>
> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
> >>
> >> at
> >>
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
> >>
> >> ... 15 more
>

[chenxyz]

Hi，Congxian：
不好意思，邮件消失在了邮件海中...
我是这么复现的，直接重启运行这个任务的TM。然后就会出现KeyedProcessFunction恢复失败。只有RocksDB StateBackend会出现这种错误，使用HDFS作为FsBackend可以正常恢复任务。一开始我以为是KeyedProcessFunction里面的自定义State恢复失败，最后写了一个空的KeyedProcessFunction也不能成功恢复任务。下面附上一个简单的Demo。
public class App {

public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

env.enableCheckpointing(2 * 60 * 1000);

DataStreamSource<Student> source = env.addSource(new SourceFunction<Student>() {

private volatile boolean running = true;

@Override
public void run(SourceContext<Student> ctx) throws Exception {
                Random rand = new Random();
                for (int i = 0; i < 100; i++) {
int id = rand.nextInt();
ctx.collect(new Student(id, "Tom" + id));
}
synchronized (this) {
while (running) {
this.wait();
}
                }
            }

@Override
public void cancel() {
synchronized (this) {
running = false;
                    this.notifyAll();
}
            }
        });

source.keyBy("id").process(new KeyedProcessFunction<Tuple, Student, Student>() {
@Override
public void processElement(Student value, Context ctx, Collector<Student> out) throws Exception {
                out.collect(value);
}
        }).addSink(new SinkFunction<Student>() {
@Override
public void invoke(Student value, Context context) throws Exception {
                System.out.println(value);
}
        });


env.execute("test keyed process operator state restore....");
}

@Getter
    @Setter
    @NoArgsConstructor
    @AllArgsConstructor
public static class Student implements Serializable {
private static final long serialVersionUID = 3909702675393996601L;
        private Integer id;
        private String name;
}
}

下面附上开启了DEBUG的log：

2020-04-14 11:42:44,679 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Establish JobManager connection for job 6fd13de6e9c84a51425f7cc34ce94940.

2020-04-14 11:42:44,684 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Offer reserved slots to the leader of job 6fd13de6e9c84a51425f7cc34ce94940.

2020-04-14 11:42:44,727 DEBUG org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager  - Registered new allocation id ed04b5323aa885406201e85c9f8b7c78 for local state stores for job 6fd13de6e9c84a51425f7cc34ce94940.

2020-04-14 11:42:44,729 DEBUG org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager  - Registered new local state store with configuration LocalRecoveryConfig{localRecoveryMode=false, localStateDirectories=LocalRecoveryDirectoryProvider{rootDirectories=[/data/flink1_10/tmp/localState/aid_ed04b5323aa885406201e85c9f8b7c78], jobID=6fd13de6e9c84a51425f7cc34ce94940, jobVertexID=bc764cd8ddf7a0cff126f51c16239658, subtaskIndex=0}} for 6fd13de6e9c84a51425f7cc34ce94940 - bc764cd8ddf7a0cff126f51c16239658 - 0 under allocation id ed04b5323aa885406201e85c9f8b7c78.

2020-04-14 11:42:44,742 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartitionFactory  - Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc): Initialized org.apache.flink.runtime.io.network.partition.ResultPartitionFactory@41801faf

2020-04-14 11:42:44,747 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Received task Source: Custom Source (1/1).

2020-04-14 11:42:44,748 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc) switched from CREATED to DEPLOYING.

2020-04-14 11:42:44,748 INFO  org.apache.flink.runtime.taskmanager.Task                     - Creating FileSystem stream leak safety net for task Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc) [DEPLOYING]

2020-04-14 11:42:44,751 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc) [DEPLOYING].

2020-04-14 11:42:44,752 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl  - Activate slot ed04b5323aa885406201e85c9f8b7c78.

2020-04-14 11:42:44,772 DEBUG org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager  - Registered new local state store with configuration LocalRecoveryConfig{localRecoveryMode=false, localStateDirectories=LocalRecoveryDirectoryProvider{rootDirectories=[/data/flink1_10/tmp/localState/aid_ed04b5323aa885406201e85c9f8b7c78], jobID=6fd13de6e9c84a51425f7cc34ce94940, jobVertexID=20ba6b65f97481d5570070de90e4e791, subtaskIndex=0}} for 6fd13de6e9c84a51425f7cc34ce94940 - 20ba6b65f97481d5570070de90e4e791 - 0 under allocation id ed04b5323aa885406201e85c9f8b7c78.

2020-04-14 11:42:44,786 DEBUG org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateFactory  - KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d): Created 1 input channels (local: 1, remote: 0, unknown: 0).

2020-04-14 11:42:44,788 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Received task KeyedProcess -> Sink: Unnamed (1/1).

2020-04-14 11:42:44,795 INFO  org.apache.flink.runtime.taskmanager.Task                     - KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d) switched from CREATED to DEPLOYING.

2020-04-14 11:42:44,805 INFO  org.apache.flink.runtime.taskmanager.Task                     - Creating FileSystem stream leak safety net for task KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d) [DEPLOYING]

2020-04-14 11:42:44,812 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d) [DEPLOYING].

2020-04-14 11:42:44,817 DEBUG org.apache.flink.runtime.blob.FileSystemBlobStore             - Copying from hdfs://nameservice1/data/flink1_10/ha/flink1_10_0/blob/job_6fd13de6e9c84a51425f7cc34ce94940/blob_p-6581a081d862993cf5a06573dbb6621fef1e46b2-f795a9ecd636e88bdf7ddd7746b9ca06 to /data/flink1_10/tmp/blobStore-e924cf2e-5e6c-48c2-893e-c2e9c0a809b6/incoming/temp-00000000.

2020-04-14 11:42:45,060 DEBUG org.apache.flink.runtime.taskexecutor.TaskExecutor            - Received heartbeat request from 4ff74d2e5ff4f66a88688fdeafd2d3ec.

2020-04-14 11:42:45,919 DEBUG org.apache.flink.runtime.taskmanager.Task                     - Getting user code class loader for task ee17273414060c57d2d331a83d1a84fc at library cache manager took 1167 milliseconds

2020-04-14 11:42:45,920 DEBUG org.apache.flink.runtime.taskmanager.Task                     - Getting user code class loader for task 406f2d0b26fb4b1040ae5ac00028202d at library cache manager took 1108 milliseconds

2020-04-14 11:42:45,931 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc) [DEPLOYING].

2020-04-14 11:42:45,931 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d) [DEPLOYING].

2020-04-14 11:42:45,934 DEBUG org.apache.flink.runtime.io.network.buffer.LocalBufferPool    - Using a local buffer pool with 2-10 buffers

2020-04-14 11:42:45,934 DEBUG org.apache.flink.runtime.io.network.buffer.LocalBufferPool    - Using a local buffer pool with 0-8 buffers

2020-04-14 11:42:45,935 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartitionManager  - Registered ReleaseOnConsumptionResultPartition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc [PIPELINED_BOUNDED, 1 subpartitions, 1 pending consumptions].

2020-04-14 11:42:45,935 DEBUG org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel  - LocalInputChannel [0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc]: Requesting LOCAL subpartition 0 of partition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc.

2020-04-14 11:42:45,935 DEBUG org.apache.flink.runtime.io.network.TaskEventDispatcher       - registering 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc

2020-04-14 11:42:45,935 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartitionManager  - Requesting subpartition 0 of ReleaseOnConsumptionResultPartition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc [PIPELINED_BOUNDED, 1 subpartitions, 1 pending consumptions].

2020-04-14 11:42:45,935 DEBUG org.apache.flink.runtime.io.network.partition.PipelinedSubpartition  - Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc): Creating read view for subpartition 0 of partition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc.

2020-04-14 11:42:45,937 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartition  - Created PipelinedSubpartitionView(index: 0) of ResultPartition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc

2020-04-14 11:42:45,961 INFO  org.apache.flink.runtime.taskmanager.Task                     - KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d) switched from DEPLOYING to RUNNING.

2020-04-14 11:42:45,963 DEBUG org.apache.flink.streaming.runtime.tasks.StreamTask           - Initializing KeyedProcess -> Sink: Unnamed (1/1).

2020-04-14 11:42:45,967 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory

2020-04-14 11:42:45,984 DEBUG org.apache.flink.streaming.runtime.tasks.StreamTask           - Using partitioner HASH for output 0 of task Source: Custom Source

2020-04-14 11:42:45,992 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using predefined options: DEFAULT.

2020-04-14 11:42:45,992 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.

2020-04-14 11:42:45,994 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc) switched from DEPLOYING to RUNNING.

2020-04-14 11:42:45,994 DEBUG org.apache.flink.streaming.runtime.tasks.StreamTask           - Initializing Source: Custom Source (1/1).

2020-04-14 11:42:45,994 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory

2020-04-14 11:42:45,995 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using predefined options: DEFAULT.

2020-04-14 11:42:45,995 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.

2020-04-14 11:42:46,033 DEBUG org.apache.flink.streaming.runtime.tasks.StreamTask           - Invoking Source: Custom Source (1/1)

2020-04-14 11:42:46,042 DEBUG org.apache.flink.streaming.api.operators.BackendRestorerProcedure  - Creating operator state backend for StreamSource_bc764cd8ddf7a0cff126f51c16239658_(1/1) with empty state.

2020-04-14 11:42:46,057 DEBUG org.apache.flink.streaming.runtime.tasks.StreamTask           - Invoking KeyedProcess -> Sink: Unnamed (1/1)

2020-04-14 11:42:46,060 DEBUG org.apache.flink.runtime.state.TaskStateManagerImpl           - Operator c09dc291fad93d575e015871097bfc60 has remote state SubtaskState{operatorStateFromBackend=StateObjectCollection{[]}, operatorStateFromStream=StateObjectCollection{[]}, keyedStateFromBackend=StateObjectCollection{[]}, keyedStateFromStream=StateObjectCollection{[]}, stateSize=0} from job manager and local state alternatives [] from local state store org.apache.flink.runtime.state.NoOpTaskLocalStateStoreImpl@3fd9cf09.

2020-04-14 11:42:46,060 DEBUG org.apache.flink.streaming.api.operators.BackendRestorerProcedure  - Creating operator state backend for StreamSink_c09dc291fad93d575e015871097bfc60_(1/1) with empty state.

2020-04-14 11:42:46,069 DEBUG org.apache.flink.runtime.state.TaskStateManagerImpl           - Operator 20ba6b65f97481d5570070de90e4e791 has remote state SubtaskState{operatorStateFromBackend=StateObjectCollection{[]}, operatorStateFromStream=StateObjectCollection{[]}, keyedStateFromBackend=StateObjectCollection{[IncrementalRemoteKeyedStateHandle{backendIdentifier=04ac09d6-1f1f-4a6c-a78d-74090c83b3c7, keyGroupRange=KeyGroupRange{startKeyGroup=0, endKeyGroup=127}, checkpointId=1, sharedState={}, privateState={MANIFEST-000006=ByteStreamStateHandle{handleName='hdfs://nameservice1/data/flink1_10/checkpoint/6fd13de6e9c84a51425f7cc34ce94940/shared/2ff261b8-f51c-42bf-9fab-93c6b119dcff', dataBytes=206}, OPTIONS-000010=File State: hdfs://nameservice1/data/flink1_10/checkpoint/6fd13de6e9c84a51425f7cc34ce94940/shared/426c66a6-d32e-43c8-9873-550237ee0963 [10379 bytes], CURRENT=ByteStreamStateHandle{handleName='hdfs://nameservice1/data/flink1_10/checkpoint/6fd13de6e9c84a51425f7cc34ce94940/shared/bbbce7c9-ea02-4590-9b18-d7a322deb2f4', dataBytes=16}}, metaStateHandle=File State: hdfs://nameservice1/data/flink1_10/checkpoint/6fd13de6e9c84a51425f7cc34ce94940/chk-1/9215630d-632e-48f6-b668-7dc235a8ff7a [1163 bytes], registered=false}]}, keyedStateFromStream=StateObjectCollection{[]}, stateSize=11764} from job manager and local state alternatives [] from local state store org.apache.flink.runtime.state.NoOpTaskLocalStateStoreImpl@3fd9cf09.

2020-04-14 11:42:46,070 DEBUG org.apache.flink.streaming.api.operators.BackendRestorerProcedure  - Creating keyed state backend for KeyedProcessOperator_20ba6b65f97481d5570070de90e4e791_(1/1) and restoring with state from alternative (1/1).

2020-04-14 11:42:46,071 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Attempting to load RocksDB native library and store it under '/data/flink1_10/tmp'

2020-04-14 11:42:46,071 DEBUG org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Attempting to create RocksDB native library folder /data/flink1_10/tmp/rocksdb-lib-a5f35d4dd06539876a20dbabc82a7f33

2020-04-14 11:42:46,078 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf  - -Dorg.apache.flink.shaded.netty4.io.netty.buffer.checkAccessible: true

2020-04-14 11:42:46,079 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf  - -Dorg.apache.flink.shaded.netty4.io.netty.buffer.checkBounds: true

2020-04-14 11:42:46,080 DEBUG org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetectorFactory  - Loaded default ResourceLeakDetector: org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector@28a9bbee

2020-04-14 11:42:46,150 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Successfully loaded RocksDB native library

2020-04-14 11:42:46,154 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Getting managed memory shared cache for RocksDB.

2020-04-14 11:42:46,161 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Obtained shared RocksDB cache of size 53687092 bytes

2020-04-14 11:42:46,495 DEBUG org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation  - Restoring keyed backend uid in operator KeyedProcessOperator_20ba6b65f97481d5570070de90e4e791_(1/1) from incremental snapshot to 04ac09d6-1f1f-4a6c-a78d-74090c83b3c7.

2020-04-14 11:42:46,571 ERROR org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder  - Caught unexpected exception.

java.nio.file.NoSuchFileException: /data/flink1_10/tmp/flink-io-cb7e58a9-8b06-4ada-b8eb-ad754cd2470e/job_6fd13de6e9c84a51425f7cc34ce94940_op_KeyedProcessOperator_20ba6b65f97481d5570070de90e4e791__1_1__uuid_0079fc9d-30c1-452a-86b9-eaa71eab7593/2508f380-d4ae-4316-9ec8-dd13ae5d3c2e/CURRENT

        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

        at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)

        at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)

        at java.nio.file.Files.copy(Files.java:1274)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:483)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)

        at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)

        at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)

        at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

        at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)

        at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)

        at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)

        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)

        at java.lang.Thread.run(Thread.java:748)

2020-04-14 11:42:46,576 WARN  org.apache.flink.streaming.api.operators.BackendRestorerProcedure  - Exception while restoring keyed state backend for KeyedProcessOperator_20ba6b65f97481d5570070de90e4e791_(1/1) from alternative (1/1), will retry while more alternatives are available.

org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected exception.

        at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)

        at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)

        at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

        at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)

        at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)

        at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)

        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.nio.file.NoSuchFileException: /data/flink1_10/tmp/flink-io-cb7e58a9-8b06-4ada-b8eb-ad754cd2470e/job_6fd13de6e9c84a51425f7cc34ce94940_op_KeyedProcessOperator_20ba6b65f97481d5570070de90e4e791__1_1__uuid_0079fc9d-30c1-452a-86b9-eaa71eab7593/2508f380-d4ae-4316-9ec8-dd13ae5d3c2e/CURRENT

        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

        at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)

        at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)

        at java.nio.file.Files.copy(Files.java:1274)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:483)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)

        at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)

        ... 15 more

2020-04-14 11:42:46,579 INFO  org.apache.flink.runtime.taskmanager.Task                     - KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d) switched from RUNNING to FAILED.

java.lang.Exception: Exception while creating StreamOperatorStateContext.

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:191)

        at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)

        at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)

        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)

        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)

        at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_20ba6b65f97481d5570070de90e4e791_(1/1) from any of the 1 provided restore options.

        at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)

        ... 9 more

Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected exception.

        at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)

        at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)

        at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)

        at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

        at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

        ... 11 more

Caused by: java.nio.file.NoSuchFileException: /data/flink1_10/tmp/flink-io-cb7e58a9-8b06-4ada-b8eb-ad754cd2470e/job_6fd13de6e9c84a51425f7cc34ce94940_op_KeyedProcessOperator_20ba6b65f97481d5570070de90e4e791__1_1__uuid_0079fc9d-30c1-452a-86b9-eaa71eab7593/2508f380-d4ae-4316-9ec8-dd13ae5d3c2e/CURRENT

        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

        at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)

        at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)

        at java.nio.file.Files.copy(Files.java:1274)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:483)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)

        at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)

        at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)

        ... 15 more

2020-04-14 11:42:46,588 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d).

2020-04-14 11:42:46,589 DEBUG org.apache.flink.runtime.taskmanager.Task                     - Release task KeyedProcess -> Sink: Unnamed (1/1) network resources (state: FAILED).

2020-04-14 11:42:46,589 DEBUG org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate  - KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d): Releasing org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate@23ae29cd.

2020-04-14 11:42:46,589 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartition  - ReleaseOnConsumptionResultPartition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc [PIPELINED_BOUNDED, 1 subpartitions, 0 pending consumptions]: Received consumed notification for subpartition 0.

2020-04-14 11:42:46,589 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartitionManager  - Received consume notification from ReleaseOnConsumptionResultPartition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc [PIPELINED_BOUNDED, 1 subpartitions, 0 pending consumptions].

2020-04-14 11:42:46,589 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartition  - Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc): Releasing ReleaseOnConsumptionResultPartition 0d224b8294583b8fcdf469150870d2a4@ee17273414060c57d2d331a83d1a84fc [PIPELINED_BOUNDED, 1 subpartitions, 0 pending consumptions].

2020-04-14 11:42:46,590 DEBUG org.apache.flink.runtime.io.network.partition.PipelinedSubpartition  - Source: Custom Source (1/1) (ee17273414060c57d2d331a83d1a84fc): Released PipelinedSubpartition#0 [number of buffers: 1 (0 bytes), number of buffers in backlog: 1, finished? false, read view? false].

2020-04-14 11:42:46,590 DEBUG org.apache.flink.runtime.io.network.partition.ResultPartitionManager  - Released partition 0d224b8294583b8fcdf469150870d2a4 produced by ee17273414060c57d2d331a83d1a84fc.

2020-04-14 11:42:46,590 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task KeyedProcess -> Sink: Unnamed (1/1) (406f2d0b26fb4b1040ae5ac00028202d) [FAILED]

2020-04-14 11:42:46,603 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state FAILED to JobManager for task KeyedProcess -> Sink: Unnamed (1/1) 406f2d0b26fb4b1040ae5ac00028202d.











在 2020-04-03 18:09:19，"Congxian Qiu" <[hidden email]> 写道：

>HDFS 上的路径和本地不一样，如果你要看 HDFS 路径的话，可能需要看 Checkjpoint Meta 的相关信息，这个比较麻烦，可以参考
>CheckpointMetadataLoadingTest 的相关测试。
>我再看了一下你给的 TM Log，看上去是 148 行的 outputStream.close() 出错了（有个比较奇怪的现象是，这里的
>outputStream 是本地的文件，但是从错误栈看是 HadoopFileSystem）。你这个是稳定复现的问题吗？如果是的话，能否贴一下打开
>debug log，贴一下 JM/TM log，另外能给一个可复现的 作业更好
>
>Best,
>Congxian
>
>
>chenxyz <[hidden email]> 于2020年4月1日周三 下午5:18写道：
>
>> Hi, 从贤，
>> 我查看了下HDFS,
>> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160下面是空的，也没有db这一层目录。
>>
>>
>>
>>
>>
>>
>>
>>
>> 在 2020-04-01 16:50:13，"Congxian Qiu" <[hidden email]> 写道：
>> >Hi
>> >Restore 可以大致分为两部分，1）下载文件；2）从下载的文件恢复
>> >从 TM 日志看像下载出错了，你可以看下
>>
>> >/data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
>> >这个文件是不是存在 double check 下，如果是下载失败，你需要确认下下载失败的原因
>> >
>> >Best,
>> >Congxian
>> >
>> >
>> >chenxyz <[hidden email]> 于2020年4月1日周三 下午3:02写道：
>> >
>> >> 任务启用rocksdb作为状态后端，任务出现异常重启时经常失败Could not restore keyed state backend for
>> >> KeyedProcessOperator。这个问题怎么解决呢？
>> >>
>> >> 版本：1.10 standalone
>> >>
>> >> 配置信息：
>> >>
>> >> state.backend: rocksdb
>> >>
>> >> state.checkpoints.dir: hdfs://nameservice1/data/flink1_10/checkpoint
>> >>
>> >> state.backend.incremental: true
>> >>
>> >> jobmanager.execution.failover-strategy: region
>> >>
>> >> io.tmp.dirs: /data/flink1_10/tmp
>> >>
>> >>
>> >>
>> >>
>> >> 任务的checkpoint配置：
>> >>
>> >> env.enableCheckpointing(2 * 60 * 1000);
>> >>
>> >>
>> >>
>> env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
>> >>
>> >> env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2 * 60 * 1000);
>> >>
>> >> env.getCheckpointConfig().setCheckpointTimeout(60000);
>> >>
>> >> env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
>> >>
>> >> env.getCheckpointConfig().setTolerableCheckpointFailureNumber(12);
>> >>
>> >>
>> >>
>> env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
>> >>
>> >>
>> >>
>> >>
>> >> 日志信息：
>> >>
>> >>
>> >>
>> >>
>> >> 2020-04-01 11:13:03
>> >>
>> >> java.lang.Exception: Exception while creating
>> StreamOperatorStateContext.
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:191)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>> >>
>> >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>> >>
>> >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>> >>
>> >> at java.lang.Thread.run(Thread.java:748)
>> >>
>> >> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
>> >> state backend for
>> >> KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926_(1/2) from any of
>> the
>> >> 1 provided restore options.
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>> >>
>> >> ... 9 more
>> >>
>> >> Caused by: org.apache.flink.runtime.state.BackendBuildingException:
>> Caught
>> >> unexpected exception.
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>> >>
>> >> ... 11 more
>> >>
>> >> Caused by: java.nio.file.NoSuchFileException:
>> >>
>> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/db/001888.sst
>> >> ->
>> >>
>> /data/flink1_10/tmp/flink-io-01229972-48d4-4229-ac8c-33f0edfe5b7c/job_5ec178dc885a8f1a64c1925e182562e3_op_KeyedProcessOperator_da2b90ef97e5c844980791c8fe08b926__1_2__uuid_772b4663-f633-4ed5-a67a-d1904760a160/3e979cc5-82c1-42bf-a269-1ce6e43f3e10/001888.sst
>> >>
>> >> at
>> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>> >>
>> >> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>> >>
>> >> at
>> >>
>> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
>> >>
>> >> at java.nio.file.Files.createLink(Files.java:1086)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreInstanceDirectoryFromPath(RocksDBIncrementalRestoreOperation.java:480)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:218)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:194)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>> >>
>> >> ... 15 more
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> TaskManager的报错信息：
>> >>
>> >>
>> >>
>> >>
>> >> 2020-04-01 14:48:10,726 ERROR
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder  -
>> >> Caught unexpected exception.
>> >>
>> >> java.io.InterruptedIOException: Interrupted while waiting for data to
>> be
>> >> acknowledged by pipeline
>> >>
>> >> at
>> >>
>> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
>> >>
>> >> at
>> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
>> >>
>> >> at
>> >>
>> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
>> >>
>> >> at
>> >>
>> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
>> >>
>> >> at
>> >>
>> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
>> >>
>> >> at
>> >>
>> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>> >>
>> >> at
>> >>
>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
>> >>
>> >> at
>> >>
>> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
>> >>
>> >> at
>> >>
>> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>> >>
>> >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>> >>
>> >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>> >>
>> >> at java.lang.Thread.run(Thread.java:748)
>> >>
>> >> 2020-04-01 14:48:10,726 WARN
>> >> org.apache.flink.streaming.api.operators.BackendRestorerProcedure  -
>> >> Exception while restoring keyed state backend for
>> >> KeyedProcessOperator_0cead2d40df6e304d3168f6366c79e3f_(1/2) from
>> >> alternative (1/1), will retry while more alternatives are available.
>> >>
>> >> org.apache.flink.runtime.state.BackendBuildingException: Caught
>> unexpected
>> >> exception.
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:336)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:548)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:288)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:304)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:131)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:255)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
>> >>
>> >> at
>> >>
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
>> >>
>> >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
>> >>
>> >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
>> >>
>> >> at java.lang.Thread.run(Thread.java:748)
>> >>
>> >> Caused by: java.io.InterruptedIOException: Interrupted while waiting
>> for
>> >> data to be acknowledged by pipeline
>> >>
>> >> at
>> >>
>> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2147)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2128)
>> >>
>> >> at
>> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2229)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
>> >>
>> >> at
>> >>
>> org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
>> >>
>> >> at
>> >>
>> org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForStateHandle(RocksDBStateDownloader.java:148)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnables$0(RocksDBStateDownloader.java:109)
>> >>
>> >> at
>> >>
>> org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:50)
>> >>
>> >> at
>> >>
>> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>> >>
>> >> at
>> >>
>> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:211)
>> >>
>> >> at
>> >>
>> java.util.concurrent.CompletableFuture.asyncRunStage(CompletableFuture.java:1640)
>> >>
>> >> at
>> >>
>> java.util.concurrent.CompletableFuture.runAsync(CompletableFuture.java:1858)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.downloadDataForAllStateHandles(RocksDBStateDownloader.java:83)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.transferAllStateDataToDirectory(RocksDBStateDownloader.java:67)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.transferRemoteStateToLocalDirectory(RocksDBIncrementalRestoreOperation.java:230)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:195)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:168)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:154)
>> >>
>> >> at
>> >>
>> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:279)
>> >>
>> >> ... 15 more
>>

[Lori681]

New » Free Games » Private Sex Orgy » Self Teen Girls
Private Video Collection » Very Explicit Cams
18 yo Teens Only » Asian Tiktok Teens
Home Made Model TKM » Real Life Cam

Download from Xubster-com:
Link: https://xubster.com/users/546/9802

Download from Daofile-com:
Link: https://daofile.com/go/3w4soyhvuake

Download from Nelion-me:
Link: https://nelion.me/go/w98s182gorax

Download from File-al:
Link: https://file.al/public/56284/31885

***************************
***************************

Log In or Sign Up; Link: https://xubster.com/free546.html

Amateur Young Girls
Link; 1: https://xubster.com/users/546/12421/0001
Link; 2: https://xubster.com/users/546/12462/0002
Link; 3: https://xubster.com/users/546/12463/0003
Link; 4: https://xubster.com/users/546/12464/0004
Link; 5: https://xubster.com/users/546/12465/0005
Link; 6: https://xubster.com/users/546/12466/0006
Link; 7: https://xubster.com/users/546/12467/0007
Link; 8: https://xubster.com/users/546/12468/0008
Link; 9: https://xubster.com/users/546/12469/0009
Link; 10: https://xubster.com/users/546/12470/0010
Link; 11: https://xubster.com/users/546/12471/0011
Link; 12: https://xubster.com/users/546/12472/0012

Covid 17-19-21-25 Girls
Link; 13: https://xubster.com/users/546/12422/0013
Link; 14: https://xubster.com/users/546/12473/0014

18 yo Teens Only
Innocent sweethearts star in 18 year old sex scenes
with their tight cunts filled by big cocks,
Sexy coeds give hot blowjobs and swallow cum,
Teens, Hardcore, Large Shlong, Amateur eighteen Years Old,
18 year old masturbates on cam,
Horny Teen 19 Years Old Masturbating On Web Cam,
Only fingers filling my pussy to orgasm.
Link; 15: https://xubster.com/users/546/12423/0015
Link; 16: https://xubster.com/users/546/12474/0016
Link; 17: https://xubster.com/users/546/12475/0017
Link; 18: https://xubster.com/users/546/12476/0018
Link; 19: https://xubster.com/users/546/12477/0019

Asian Tiktok Teens
Link; 20: https://xubster.com/users/546/12424/0020
Link; 21: https://xubster.com/users/546/12479/0021
Link; 22: https://xubster.com/users/546/12480/0022
Link; 23: https://xubster.com/users/546/12481/0023
Link; 24: https://xubster.com/users/546/12482/0024
Link; 25: https://xubster.com/users/546/12483/0025
Link; 26: https://xubster.com/users/546/12484/0026
Link; 27: https://xubster.com/users/546/12485/0027
Link; 28: https://xubster.com/users/546/12486/0028
Link; 29: https://xubster.com/users/546/12487/0029
Link; 30: https://xubster.com/users/546/12488/0030
Link; 31: https://xubster.com/users/546/12489/0031

Real Life Cam
Teen and Young Girls and Couples,
Voyeur Villa Nelly Doggystyle HD Sex,
Real Private Life on WebCam,
Categories: Voyeur Sex, WebCam Porn, SpyCam Fetish,
Teen Cam, Young Couples, Exhibitionism Video, Young Public Sex,
WebCam Public Sex, Masturbation Teens, TeenSex
Link; 32: https://xubster.com/users/546/12418/0032
Link; 33: https://xubster.com/users/546/12490/0033
Link; 34: https://xubster.com/users/546/12491/0034
Link; 35: https://xubster.com/users/546/12492/0035
Link; 36: https://xubster.com/users/546/12493/0036
Link; 37: https://xubster.com/users/546/12494/0037

Porn Tiktok 18+ Banned on TikTok
Link; 38: https://xubster.com/users/546/12592/0038
Link; 39: https://xubster.com/users/546/12593/0039
Link; 40: https://xubster.com/users/546/12594/0040
Link; 41: https://xubster.com/users/546/12595/0041
Link; 42: https://xubster.com/users/546/12596/0042
Link; 43: https://xubster.com/users/546/12597/0043
Link; 44: https://xubster.com/users/546/12598/0044
Link; 45: https://xubster.com/users/546/12599/0045
Link; 46: https://xubster.com/users/546/12600/0046
Link; 47: https://xubster.com/users/546/12601/0047
Link; 48: https://xubster.com/users/546/12602/0048
Link; 49: https://xubster.com/users/546/12603/0049
Link; 50: https://xubster.com/users/546/12604/0050

Teen Models
Japanese Teen Girl in WebCam Show After School
MISS VIKKI - TEEN RUSSIAN MODEL
Little Miss Vikki From Russia – My Private Collection
Mattie Doll – Horny Teen With a Talent for Sharing Sensational Orgasmes
Kyutty Kitty – Asian Sweety PussyCat
REAL VIDEOS OF SEXY TEEN MODEL Hentai-Cat
Effy Loweell – Sexy Young Models With Small Tits

Alice MFC
Link; 51: https://xubster.com/users/546/12427

Alison Lil Baby
Link; 52: https://xubster.com/users/546/12428

Cute Mary
Link; 53: https://xubster.com/users/546/12429

Effy Loweell
Link; 54: https://xubster.com/users/546/12430

Hana Lily
Link; 55: https://xubster.com/users/546/12431

Hentai-Cat
Link; 56: https://xubster.com/users/546/12432

Hot Nesquik
Link; 57: https://xubster.com/users/546/12433

Katya Letova
Link; 58: https://xubster.com/users/546/12434

Koska Leska
Link; 59: https://xubster.com/users/546/12435

Kyutty
Link; 60: https://xubster.com/users/546/12436

Mattie Doll
Link; 61: https://xubster.com/users/546/12437

Miss Vikki
Link; 62: https://xubster.com/users/546/12438

Venus Kitty
Link; 63: https://xubster.com/users/546/12439

Your Wet Schoolgirl
Link; 64: https://xubster.com/users/546/12440

Non Nude Tiktok Teens
Teen Cute Girls talk, sexy dance and play on cam
Link; 65: https://xubster.com/users/546/12452/0065
Link; 66: https://xubster.com/users/546/12507/0066
Link; 67: https://xubster.com/users/546/12508/0067
Link; 68: https://xubster.com/users/546/12509/0068
Link; 69: https://xubster.com/users/546/12510/0069
Link; 70: https://xubster.com/users/546/12511/0070
Link; 71: https://xubster.com/users/546/12512/0071
Link; 72: https://xubster.com/users/546/12513/0072
Link; 73: https://xubster.com/users/546/12514/0073
Link; 74: https://xubster.com/users/546/12515/0074
Link; 75: https://xubster.com/users/546/12516/0075
Link; 76: https://xubster.com/users/546/12517/0076

Nudism Young Girls
Link; 77: https://xubster.com/users/546/12453/0077
Link; 78: https://xubster.com/users/546/12518/0078
Link; 79: https://xubster.com/users/546/12519/0079
Link; 80: https://xubster.com/users/546/12520/0080
Link; 81: https://xubster.com/users/546/12521/0081
Link; 82: https://xubster.com/users/546/12522/0082
Link; 83: https://xubster.com/users/546/12523/0083

Russian Family Incest
Incest family teens
Link; 84: https://xubster.com/users/546/12454/0084
Link; 85: https://xubster.com/users/546/12524/0085
Link; 86: https://xubster.com/users/546/12525/0086
Link; 87: https://xubster.com/users/546/12526/0087
Link; 88: https://xubster.com/users/546/12527/0088
Link; 89: https://xubster.com/users/546/12528/0089
Link; 90: https://xubster.com/users/546/12529/0090
Link; 91: https://xubster.com/users/546/12530/0091

Real Spycam - Hiddencam
Link; 92: https://xubster.com/users/546/12455/0092
Link; 93: https://xubster.com/users/546/12531/0093
Link; 94: https://xubster.com/users/546/12532/0094
Link; 95: https://xubster.com/users/546/12533/0095
Link; 96: https://xubster.com/users/546/12534/0096
Link; 97: https://xubster.com/users/546/12535/0097
Link; 98: https://xubster.com/users/546/12536/0098
Link; 99: https://xubster.com/users/546/12537/0099
Link; 100: https://xubster.com/users/546/12538/0100
Link; 101: https://xubster.com/users/546/12539/0101
Link; 102: https://xubster.com/users/546/12540/0102
Link; 103: https://xubster.com/users/546/12541/0103

Tight Teen Pussy
FUCK TIGHT TEEN PUSSY - Real Fuck Extreme Small Teen Pussy - 18+
FULL HD 83 Hot Home Made Videos of Real Extreme Fuck Small Teen Pussy
Link; 104: https://xubster.com/users/546/12456/0104

Random Tiktok Girls
Link; 105: https://xubster.com/users/546/12457/0105
Link; 106: https://xubster.com/users/546/12542/0106
Link; 107: https://xubster.com/users/546/12543/0107
Link; 108: https://xubster.com/users/546/12544/0108
Link; 109: https://xubster.com/users/546/12545/0109
Link; 110: https://xubster.com/users/546/12546/0110
Link; 111: https://xubster.com/users/546/12547/0111
Link; 112: https://xubster.com/users/546/12548/0112
Link; 113: https://xubster.com/users/546/12549/0113
Link; 114: https://xubster.com/users/546/12550/0114
Link; 115: https://xubster.com/users/546/12551/0115
Link; 116: https://xubster.com/users/546/12552/0116
Link; 117: https://xubster.com/users/546/12553/0117
Link; 118: https://xubster.com/users/546/12554/0118

Skype and Omegle Girls
Link; 119: https://xubster.com/users/546/12459/0119
Link; 120: https://xubster.com/users/546/12555/0120
Link; 121: https://xubster.com/users/546/12556/0121
Link; 122: https://xubster.com/users/546/12557/0122
Link; 123: https://xubster.com/users/546/12558/0123
Link; 124: https://xubster.com/users/546/12559/0124
Link; 125: https://xubster.com/users/546/12560/0125
Link; 126: https://xubster.com/users/546/12561/0126
Link; 127: https://xubster.com/users/546/12562/0127
Link; 128: https://xubster.com/users/546/12563/0128
Link; 129: https://xubster.com/users/546/12564/0129
Link; 130: https://xubster.com/users/546/12565/0130

Tiktok Nude Girls
Link; 131: https://xubster.com/users/546/12460/0131
Link; 132: https://xubster.com/users/546/12566/0132
Link; 133: https://xubster.com/users/546/12567/0133
Link; 134: https://xubster.com/users/546/12568/0134
Link; 135: https://xubster.com/users/546/12569/0135
Link; 136: https://xubster.com/users/546/12570/0136
Link; 137: https://xubster.com/users/546/12571/0137
Link; 138: https://xubster.com/users/546/12572/0138
Link; 139: https://xubster.com/users/546/12573/0139
Link; 140: https://xubster.com/users/546/12574/0140
Link; 141: https://xubster.com/users/546/12575/0141
Link; 142: https://xubster.com/users/546/12576/0142
Link; 143: https://xubster.com/users/546/12577/0143
Link; 144: https://xubster.com/users/546/12578/0144
Link; 145: https://xubster.com/users/546/12579/0145

Webcam Teens and Couples
Link; 146: https://xubster.com/users/546/12461/0146
Link; 147: https://xubster.com/users/546/12580/0147
Link; 148: https://xubster.com/users/546/12581/0148
Link; 149: https://xubster.com/users/546/12582/0149
Link; 150: https://xubster.com/users/546/12583/0150
Link; 151: https://xubster.com/users/546/12584/0151
Link; 152: https://xubster.com/users/546/12585/0152
Link; 153: https://xubster.com/users/546/12586/0153
Link; 154: https://xubster.com/users/546/12587/0154
Link; 155: https://xubster.com/users/546/12588/0155
Link; 156: https://xubster.com/users/546/12589/0156
Link; 157: https://xubster.com/users/546/12590/0157

Rape Porn; Free Porn Videos; HD - VR Sex Videos
Link; 158: https://xubster.com/users/546/6261/001

Snuff Porn Videos; Fake murders and decapitation
Link; 159: https://xubster.com/users/546/6265/002

Forced Porn; Forced Sex - Forced To Fuck Videos
Link; 160: https://xubster.com/users/546/6266/003

Hot Asian Teen Sex Videos; Japanese and Korean Porn Movies
Link; 161: https://xubster.com/users/546/6262/004

Sleeping Girl Gets An Unexpected Visit Late At Night
Sex Sleeping Girl Porn Videos
Link; 162: https://xubster.com/users/546/6296/005

Lesbian Necrophilia Porn Videos
Link; 163: https://xubster.com/users/546/6290/006

Horror Porn Videos - Sex Movies
Link; 164: https://xubster.com/users/546/6295/007

Sex Gay Porn Videos
Link; 165: https://xubster.com/users/546/6281/008

:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

3D, Hentai, Free Games

Download from Daofile.com:
Link: https://daofile.com/go/p47ssveuv75c

Download:
Sex Cartoons; Free Games; Hentai; Manga; 3D
MegaPack; 367249 Files

Download: MegaPack; 367249:
Link: https://file.al/public/56284/38916

:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

Preview Video
Link: https://xubster.com/users/546/12061

HD Young Nudists PN Videos

Download from Xubster.com:
Link: https://xubster.com/users/546/12062
Link: https://xubster.com/users/546/12067

Download:
PN Teen Girls; Young Adult Nudism; Teen Nudist Sex:
MegaPack; 83150 Files

Download: MegaPack; 83150:
Link: https://file.al/public/56284/38915

:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

Private Collection; Real Teen Sexy Selfies and Videos
MegaPack; 192851 Files

Download: MegaPack; 192851:
Link: https://file.al/public/56284/38917

:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

Preview Video
Link: https://daofile.com/go/efb4i0wyu73a

TukTuk Patrol - Thai Teen Video:

Download from Daofile.com:
Link: https://daofile.com/go/0ackk6rwv5gq

:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

Girls and Boys PT - Photo and Video
MegaPack; 531498 Files

Download: MegaPack; 531498:
Link: https://file.al/public/56284/39650

:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

VIP: - Young Nude Vagina
Link; 1: https://daofile.com/go/58017o3w2wa1

VIP: - Taboo Teen Archive
Link; 2: https://daofile.com/go/at6nq7tzdrwq
Link; 3: https://daofile.com/go/uqvdfvlt1b7j

VIP: - Private Sex Orgy: - Self Teen Girls
Link; 4: https://daofile.com/go/rwmcfthjrcew
Link; 5: https://daofile.com/go/7x4q0mtks6bo

Young Teen Bathing Suit Videos and HD Footage
Asian Woman In Bathroom Videos and HD Footage
Japanese Bath Culture, Public Bath
Link; 6: https://daofile.com/go/zvcjqfm0s50w
Link; 7: https://daofile.com/go/62mt4oaxq78n
Link; 8: https://daofile.com/go/x1q0iheql2ym
Link; 9: https://daofile.com/go/c9d7xs7fsme6
Link; 10: https://daofile.com/go/0pza6o1prc3r
Link; 11: https://daofile.com/go/s6qkgi10c7tf
Link; 12: https://daofile.com/go/1mg548npuj9p
Link; 13: https://daofile.com/go/piif3sxzu7y4

Japanese Teen HD Sex Porn Videos
Japanese School Girl Full Movie Porn videos
Japanese teen jav xxx sex school asian big tits milf mom sister porn HD
Link; 14: https://daofile.com/go/ws3qwum15koi
Link; 15: https://daofile.com/go/z9k8qssuw74c
Link; 16: https://daofile.com/go/r2pmntlq6vkp
Link; 17: https://daofile.com/go/r9khigu0c0xe
Link; 18: https://daofile.com/go/5qfkaafzvk0k
Link; 19: https://daofile.com/go/st4jcfg1g9bz
Link; 20: https://daofile.com/go/4hvigt8dchbc

Selfie teens
Real teens sexy selfies, show teen tits
Link; 21: https://daofile.com/go/dlfstx2s3mv3
Link; 22: https://daofile.com/go/3i181cjpm77j
Link; 23: https://daofile.com/go/ccyhj01bdnmg
Link; 24: https://daofile.com/go/gh7snep8cn54

Collection of Teen Sex and Erotic Videos
Link; 25: https://daofile.com/go/kun7aw1l0sxy
Link; 26: https://daofile.com/go/u9jikdewbmen
Link; 27: https://daofile.com/go/sxflclskqlde
Link; 28: https://daofile.com/go/htsmkg04kkop
Link; 29: https://daofile.com/go/daq9svwypcpg
Link; 30: https://daofile.com/go/d26g52rcnyql
Link; 31: https://daofile.com/go/4msqlcw96jyf
Link; 32: https://daofile.com/go/a6vapjguf0x7
Link; 33: https://daofile.com/go/937pupbznnt1

Webcam Teen
Teen Erotic Videos From Real Life Cams - Omegle teen, Skype teen
Link; 34: https://daofile.com/go/e2nnzbuhjt5z
Link; 35: https://daofile.com/go/nz1tewuygcr1
Link; 36: https://daofile.com/go/23sfxojnkhlc
Link; 37: https://daofile.com/go/31lpybl6312o
Link; 38: https://daofile.com/go/ahtftflfq6gl
Link; 39: https://daofile.com/go/5xcpj94xj6tw
Link; 40: https://daofile.com/go/gyshuzhg00l8
Link; 41: https://daofile.com/go/kr3zaonpkf4p
Link; 42: https://daofile.com/go/cl923bdxvs9k
Link; 43: https://daofile.com/go/3burlwssg7py
Link; 44: https://daofile.com/go/e12thco5doao

Young Girls and Boys Make Real Hot Sex on Cam
Link; 45: https://daofile.com/go/k0ws7lypjw5c
Link; 46: https://daofile.com/go/w96f0hj7ym8t

Sex Machine Porn Videos
Link; 47: https://daofile.com/go/psc0hbsfch2w
Link; 48: https://daofile.com/go/xet9s4b8l1n0

Teen Crazy Girls Gallery
Link; 49: https://daofile.com/go/ka68my4wdqca
Link; 50: https://daofile.com/go/wwiaf2oaavgp

Toilet HD Videos - Hidden cams in toilets film every amateur comer.
Watch Public toilet spy cam of girls pissing of Best Collection Voyeur Porn videos.
Medical And Gyno Voyeur Videos.
Watch Medical voyeur cam shooting,
Asian explored in the gyno office of Best Collection Voyeur Porn videos.
Link; 51: https://daofile.com/go/71feh10vjrfe
Link; 52: https://daofile.com/go/osxvp1epjyam
Link; 53: https://daofile.com/go/ylhuc48hu73l
Link; 54: https://daofile.com/go/5lx4gmnok82y
Link; 55: https://daofile.com/go/u13oih6vbrjc
Link; 56: https://daofile.com/go/s3a5qk1p5cyw

TokyoDolls: Sexy Teen Girls - Full Collection
Link; 57: https://daofile.com/go/y9w47mnulyw1

Galitsin teen - TP Sex Videos
Link; 58: https://daofile.com/go/6uttrr3le10n
Link; 59: https://daofile.com/go/sjatsg7tjroi
Link; 60: https://daofile.com/go/1gx4d40cf40w
Link; 61: https://daofile.com/go/mu8hlmao4fge
Link; 62: https://daofile.com/go/phu0sv1tgx9c
Link; 63: https://daofile.com/go/z6oa86xp644b

Candid HD
Link; 64: https://daofile.com/go/1piz0c48n4p3

Femdom BDSM
Link; 65: https://daofile.com/go/u6w29zcyys0c

Nonude models
Link; 66: https://daofile.com/go/nnxix879th8k

Nudi-Pageant
Link; 67: https://daofile.com/go/nxnme8zrveru

TTL and YFM Teen Latinas Models
Link; 68: https://daofile.com/go/wlpk7947rax3

****************************
****************************

New » Free Games » Private Sex Orgy » Self Teen Girls
Private Video Collection » Very Explicit Cams
18 yo Teens Only » Asian Tiktok Teens
Home Made Model TKM » Real Life Cam

Download from Xubster-com:
Link: https://xubster.com/users/546/9802

Download from Daofile-com:
Link: https://daofile.com/go/3w4soyhvuake

Download from Nelion-me:
Link: https://nelion.me/go/w98s182gorax

Download from File-al:
Link: https://file.al/public/56284/31885

¤¤
¤¤
¤¤
¤¤
¤¤