hi,
我们在使用flink-stream时遇到了如下问题,有谁可以帮助一下吗?怎么避免该问题呢? flink版本:flink-1.8.1 standalone cluster node4是我们集群中的一个slave节点,我们的任务运行了差不多1个月才出现的这个问题。这个异常导致任务全部处于失败然后恢复失败的循环中,目前不知道什么原因造成的,查看了 /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/文件夹是空的。 谢谢。 2019-09-19 16:54:34,328 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04. java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) 2019-09-19 16:54:34,330 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04 . java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506074 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) 2019-09-19 16:54:34,331 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04 . java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506075 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) 2019-09-19 16:54:34,332 ERROR org.apache.flink.runtime.blob.BlobServerConnection - GET operation failed for BLOB ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 from /node04 . java.nio.file.NoSuchFileException: /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506076 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) |
我猜测原因是blob server目录里存的blob文件被删了,所以/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073目录为空,具体可以参考如下代码:
https://github.com/apache/flink/blob/1a81982ba864d72fda3d161fc82ca1572333ba11/flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java#L129 <https://github.com/apache/flink/blob/1a81982ba864d72fda3d161fc82ca1572333ba11/flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java#L129> 有一个类似问题,你可以看一下是否有帮助: http://mail-archives.apache.org/mod_mbox/flink-user/201810.mbox/%3c784feda8-9a82-c938-8dbe-d7e8e2650916@...%3e <http://mail-archives.apache.org/mod_mbox/flink-user/201810.mbox/%3C784feda8-9a82-c938-8dbe-d7e8e2650916@...%3E> Regards, Dian > 在 2019年9月19日,下午6:02,guo liu <[hidden email]> 写道: > > hi, > 我们在使用flink-stream时遇到了如下问题,有谁可以帮助一下吗?怎么避免该问题呢? > > flink版本:flink-1.8.1 standalone cluster > > node4是我们集群中的一个slave节点,我们的任务运行了差不多1个月才出现的这个问题。这个异常导致任务全部处于失败然后恢复失败的循环中,目前不知道什么原因造成的,查看了 > /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/文件夹是空的。 > > 谢谢。 > > 2019-09-19 16:54:34,328 ERROR > org.apache.flink.runtime.blob.BlobServerConnection - GET > operation failed for BLOB > ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 > from /node04. > java.nio.file.NoSuchFileException: > /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073 > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) > at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) > at > org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) > at > org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) > at > org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) > 2019-09-19 16:54:34,330 ERROR > org.apache.flink.runtime.blob.BlobServerConnection - GET > operation failed for BLOB > ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 > from /node04 . > java.nio.file.NoSuchFileException: > /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506074 > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) > at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) > at > org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) > at > org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) > at > org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) > 2019-09-19 16:54:34,331 ERROR > org.apache.flink.runtime.blob.BlobServerConnection - GET > operation failed for BLOB > ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 > from /node04 . > java.nio.file.NoSuchFileException: > /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506075 > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) > at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) > at > org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) > at > org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) > at > org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) > 2019-09-19 16:54:34,332 ERROR > org.apache.flink.runtime.blob.BlobServerConnection - GET > operation failed for BLOB > ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28 > from /node04 . > java.nio.file.NoSuchFileException: > /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506076 > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) > at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429) > at > org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497) > at > org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231) > at > org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117) |
Free forum by Nabble | Edit this page |