Apache Flink 中文用户邮件列表

Flink BlobServerConnection NoSuchFileException

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
guo liu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Flink BlobServerConnection NoSuchFileException

guo liu
hi,
我们在使用flink-stream时遇到了如下问题，有谁可以帮助一下吗？怎么避免该问题呢？

flink版本：flink-1.8.1  standalone cluster

node4是我们集群中的一个slave节点，我们的任务运行了差不多1个月才出现的这个问题。这个异常导致任务全部处于失败然后恢复失败的循环中，目前不知道什么原因造成的，查看了
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/文件夹是空的。

谢谢。

2019-09-19 16:54:34,328 ERROR
org.apache.flink.runtime.blob.BlobServerConnection            - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04.
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
2019-09-19 16:54:34,330 ERROR
org.apache.flink.runtime.blob.BlobServerConnection            - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04 .
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506074
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
2019-09-19 16:54:34,331 ERROR
org.apache.flink.runtime.blob.BlobServerConnection            - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04 .
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506075
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
2019-09-19 16:54:34,332 ERROR
org.apache.flink.runtime.blob.BlobServerConnection            - GET
operation failed for BLOB
ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
from /node04 .
java.nio.file.NoSuchFileException:
/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506076
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
Dian Fu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Flink BlobServerConnection NoSuchFileException

Dian Fu
我猜测原因是blob server目录里存的blob文件被删了，所以/tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073目录为空，具体可以参考如下代码：
https://github.com/apache/flink/blob/1a81982ba864d72fda3d161fc82ca1572333ba11/flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java#L129 <https://github.com/apache/flink/blob/1a81982ba864d72fda3d161fc82ca1572333ba11/flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java#L129>

有一个类似问题，你可以看一下是否有帮助：
http://mail-archives.apache.org/mod_mbox/flink-user/201810.mbox/%3c784feda8-9a82-c938-8dbe-d7e8e2650916@...%3e <http://mail-archives.apache.org/mod_mbox/flink-user/201810.mbox/%3C784feda8-9a82-c938-8dbe-d7e8e2650916@...%3E>

Regards,
Dian

> 在 2019年9月19日，下午6:02，guo liu <[hidden email]> 写道：
>
> hi,
> 我们在使用flink-stream时遇到了如下问题，有谁可以帮助一下吗？怎么避免该问题呢？
>
> flink版本：flink-1.8.1  standalone cluster
>
> node4是我们集群中的一个slave节点，我们的任务运行了差不多1个月才出现的这个问题。这个异常导致任务全部处于失败然后恢复失败的循环中，目前不知道什么原因造成的，查看了
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/文件夹是空的。
>
> 谢谢。
>
> 2019-09-19 16:54:34,328 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection            - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04.
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506073
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
> 2019-09-19 16:54:34,330 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection            - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04 .
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506074
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
> 2019-09-19 16:54:34,331 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection            - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04 .
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506075
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
> 2019-09-19 16:54:34,332 ERROR
> org.apache.flink.runtime.blob.BlobServerConnection            - GET
> operation failed for BLOB
> ba2fb0924f20a093cff50b27aa8ea6c1/p-105986c82d33c1ed56863caabc5f9b6517bc16c3-2122fe23e729d1ea1d0a4f13f16d1e28
> from /node04 .
> java.nio.file.NoSuchFileException:
> /tmp/blobStore-ca0f7afe-bd98-40cc-ae36-4ad96b1ef97e/incoming/temp-03506076
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at
> org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:429)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:497)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)

Free forum by Nabble Edit this page