flink写s3问题

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

flink写s3问题

lemon
 hi各位,flink 1.11.1写s3的时候输出以下日志


2020-09-11 11:06:14,680 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=1 (max part counter=2).
2020-09-11 11:06:14,685 WARN  com.amazonaws.services.s3.internal.Mimetypes                 [] - Unable to find 'mime.types' file in classpath
2020-09-11 11:06:15,133 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=1.
2020-09-11 11:06:15,138 INFO  org.apache.flink.fs.s3.common.writer.S3Committer             [] - Committing test.txt/2020-09-11--11/part-0-0 with MPU ID 2~dD7KNpNcq4Tr95ipKxT6PSEi21Yz3oe
2020-09-11 11:07:15,143 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=2 (max part counter=2).
2020-09-11 11:07:15,447 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=2.
2020-09-11 11:08:15,463 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=3 (max part counter=2).
2020-09-11 11:08:15,651 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=3.
2020-09-11 11:09:15,671 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=4 (max part counter=2).
2020-09-11 11:09:15,994 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=4.
2020-09-11 11:10:16,078 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=5 (max part counter=2).
2020-09-11 11:10:16,304 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=5.
2020-09-11 11:11:16,317 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=6 (max part counter=2).
2020-09-11 11:11:16,497 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=6.
2020-09-11 11:12:16,508 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=7 (max part counter=2).
2020-09-11 11:12:16,643 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=7.
2020-09-11 11:13:16,665 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=8 (max part counter=2).
2020-09-11 11:13:17,031 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=8.
2020-09-11 11:14:17,045 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 checkpointing for checkpoint with id=9 (max part counter=2).
2020-09-11 11:14:17,225 INFO  org.apache.flink.streaming.api.functions.sink.filesystem.Buckets [] - Subtask 0 received completion notification for checkpoint with id=9.
显示一直在commiting,下游看不到任何文件输出,请问写s3的时候是先写在本地缓存的吗?什么条件下才会在下游s3中生成文件?