flink yarn application 提交任务出错

[casel.chen]

今天尝试使用yarn application模式（带yarn.provided.lib.dirs参数)，将$FLINK_LIB目录下的jar包上传到了hdfs，结果报了如下的错，是少了哪个jar包或配置文件吗？


org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn Application Cluster

at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:443)

at com.huifu.streamsql.launcher.yarn.YarnApplicationExecutor.doStart(YarnApplicationExecutor.java:70)

at com.huifu.streamsql.launcher.AbstractJobExecutor.start(AbstractJobExecutor.java:76)

at com.huifu.streamsql.launcher.JobCommand$1.execute(JobCommand.java:12)

at com.huifu.streamsql.launcher.SubmitJobMain.main(SubmitJobMain.java:39)

Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9000/flinkLib, expected: file:///

at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:665)

at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:630)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1437)

at org.apache.flink.yarn.YarnApplicationFileUploader.lambda$getAllFilesInProvidedLibDirs$2(YarnApplicationFileUploader.java:429)

at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedConsumer$3(FunctionUtils.java:94)

at java.util.ArrayList.forEach(ArrayList.java:1259)

at org.apache.flink.yarn.YarnApplicationFileUploader.getAllFilesInProvidedLibDirs(YarnApplicationFileUploader.java:426)

at org.apache.flink.yarn.YarnApplicationFileUploader.<init>(YarnApplicationFileUploader.java:109)

at org.apache.flink.yarn.YarnApplicationFileUploader.from(YarnApplicationFileUploader.java:354)

at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:710)

at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:558)

at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:436)

... 4 more

[Yang Wang]

这个报错应该是你HDFS的core-site.xml的配置有问题

defaultFS应该是hdfs://localhost:9000/才对，你可以检查一下

Best,
Yang

casel.chen <[hidden email]> 于2021年1月19日周二 下午6:19写道：

> 今天尝试使用yarn
> application模式（带yarn.provided.lib.dirs参数)，将$FLINK_LIB目录下的jar包上传到了hdfs，结果报了如下的错，是少了哪个jar包或配置文件吗？
>
>
> org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't
> deploy Yarn Application Cluster
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:443)
>
> at
> com.huifu.streamsql.launcher.yarn.YarnApplicationExecutor.doStart(YarnApplicationExecutor.java:70)
>
> at
> com.huifu.streamsql.launcher.AbstractJobExecutor.start(AbstractJobExecutor.java:76)
>
> at com.huifu.streamsql.launcher.JobCommand$1.execute(JobCommand.java:12)
>
> at com.huifu.streamsql.launcher.SubmitJobMain.main(SubmitJobMain.java:39)
>
> Caused by: java.lang.IllegalArgumentException: Wrong FS:
> hdfs://localhost:9000/flinkLib, expected: file:///
>
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:665)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:630)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625)
>
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
>
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1437)
>
> at
> org.apache.flink.yarn.YarnApplicationFileUploader.lambda$getAllFilesInProvidedLibDirs$2(YarnApplicationFileUploader.java:429)
>
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedConsumer$3(FunctionUtils.java:94)
>
> at java.util.ArrayList.forEach(ArrayList.java:1259)
>
> at
> org.apache.flink.yarn.YarnApplicationFileUploader.getAllFilesInProvidedLibDirs(YarnApplicationFileUploader.java:426)
>
> at
> org.apache.flink.yarn.YarnApplicationFileUploader.<init>(YarnApplicationFileUploader.java:109)
>
> at
> org.apache.flink.yarn.YarnApplicationFileUploader.from(YarnApplicationFileUploader.java:354)
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:710)
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:558)
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:436)
>
> ... 4 more
>
>
>
>