crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

徐州州
完整日志如下,执行命令bin/flink run -m yarn-cluster -yjm 660 -ytm 1024 -ys 1 -yqu xjia_queue -ynm Test_Demo02 -c  com.dwd.test_main.Test_Demo02 -Drest.port="8067" /opt/module/flink1.12/xjia_lib/xjia_shuyun-6.0.jar


WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/home/xjia/opt/module/flink1.12/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2021-01-02 14:55:11,888 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at /0.0.0.0:8032
2021-01-02 14:55:12,056 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2021-01-02 14:55:12,206 WARN  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2021-01-02 14:55:12,237 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured JobManager memory is 660 MB. YARN will allocate 1024 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 364 MB may not be used by Flink.
2021-01-02 14:55:12,238 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, slotsPerTaskManager=1}
2021-01-02 14:55:12,458 WARN  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The file system scheme is 'file'. This indicates that the specified Hadoop configuration path is wrong and the system is using the default Hadoop configuration values.The Flink YARN client needs to store its files in a distributed file system
2021-01-02 14:55:13,775 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting application master application_1609403978979_0043
2021-01-02 14:55:14,025 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted application application_1609403978979_0043
2021-01-02 14:55:14,026 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the cluster to be allocated
2021-01-02 14:55:14,029 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, current state ACCEPTED


------------------------------------------------------------
 The program finished with the following exception:


org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute sql
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
        at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242)
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971)
        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
        at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1047)
Caused by: org.apache.flink.table.api.TableException: Failed to execute sql
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:696)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:759)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:665)
        at com.dwd.test_main.Test_Demo02$.main(Test_Demo02.scala:50)
        at com.dwd.test_main.Test_Demo02.main(Test_Demo02.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:316)
        ... 11 more
Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.
        at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:460)
        at org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:70)
        at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1940)
        at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128)
        at org.apache.flink.table.planner.delegation.ExecutorBase.executeAsync(ExecutorBase.java:57)
        at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:680)
        ... 20 more
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 
Diagnostics from YARN: Application application_1609403978979_0043 failed 2 times in previous 10000 milliseconds due to AM Container for appattempt_1609403978979_0043_000002 exited with  exitCode: -1000
Failing this attempt.Diagnostics: [2021-01-02 14:55:14.203]File file:/home/xjia/.flink/application_1609403978979_0043/lib/flink-sql-parquet_2.11-1.12.0.jar does not exist
java.io.FileNotFoundException: File file:/home/xjia/.flink/application_1609403978979_0043/lib/flink-sql-parquet_2.11-1.12.0.jar does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
        at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:243)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:236)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:224)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


For more detailed output, check the application tracking page: http://hadoop31:8088/cluster/app/application_1609403978979_0043 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1609403978979_0043
        at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1078)
        at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:558)
        at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:453)
        ... 25 more
2021-01-02 14:55:14,405 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cancelling deployment from Deployment Failure Hook
2021-01-02 14:55:14,413 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at /0.0.0.0:8032
2021-01-02 14:55:14,418 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Killing YARN application
2021-01-02 14:55:14,432 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Killed application application_1609403978979_0043
2021-01-02 14:55:14,533 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deleting files in file:/home/xjia/.flink/application_1609403978979_0043.
Reply | Threaded
Open this post in threaded view
|

Re: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

冯嘉伟
hi!

java.io.FileNotFoundException: File file:/home/xjia/.flink/...
可以看出,从本地加载jar包,而不是hdfs。

我觉得可能是hadoop环境的问题,导致读取的scheme是file,使用 echo $HADOOP_CLASSPATH 检查你的环境。

Important Make sure that the HADOOP_CLASSPATH environment variable is set up
(it can be checked by running echo $HADOOP_CLASSPATH). If not, set it up
using

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html>  



--
Sent from: http://apache-flink.147419.n8.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

zhisheng
hi

可以检查一下提交任务的 flink 客户端的 lib 目录下面是否有 flink-sql-parquet_2.11-1.12.0.jar 依赖

Best
zhisheng

冯嘉伟 <[hidden email]> 于2021年1月4日周一 上午9:58写道:

> hi!
>
> java.io.FileNotFoundException: File file:/home/xjia/.flink/...
> 可以看出,从本地加载jar包,而不是hdfs。
>
> 我觉得可能是hadoop环境的问题,导致读取的scheme是file,使用 echo $HADOOP_CLASSPATH 检查你的环境。
>
> Important Make sure that the HADOOP_CLASSPATH environment variable is set
> up
> (it can be checked by running echo $HADOOP_CLASSPATH). If not, set it up
> using
>
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html
> <
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html>
>
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

回复: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

徐州州
我也错在这种问题-(通过crontab启动flink-job失败),使用azkaban调度完美解决。



------------------&nbsp;原始邮件&nbsp;------------------
发件人: "zhisheng"<[hidden email]&gt;;
发送时间: 2021年1月5日(星期二) 晚上7:13
收件人: "user-zh"<[hidden email]&gt;;
主题: Re: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist



hi

可以检查一下提交任务的 flink 客户端的 lib 目录下面是否有 flink-sql-parquet_2.11-1.12.0.jar 依赖

Best
zhisheng

冯嘉伟 <[hidden email]&gt; 于2021年1月4日周一 上午9:58写道:

&gt; hi!
&gt;
&gt; java.io.FileNotFoundException: File file:/home/xjia/.flink/...
&gt; 可以看出,从本地加载jar包,而不是hdfs。
&gt;
&gt; 我觉得可能是hadoop环境的问题,导致读取的scheme是file,使用 echo $HADOOP_CLASSPATH 检查你的环境。
&gt;
&gt; Important Make sure that the HADOOP_CLASSPATH environment variable is set
&gt; up
&gt; (it can be checked by running echo $HADOOP_CLASSPATH). If not, set it up
&gt; using
&gt;
&gt;
&gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html
&gt; <
&gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html&gt;;
&gt;
&gt;
&gt;
&gt;
&gt; --
&gt; Sent from: http://apache-flink.147419.n8.nabble.com/
&gt;
Reply | Threaded
Open this post in threaded view
|

回复: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

徐州州
In reply to this post by zhisheng
我的$HADOOP_CLASSPATH指向的这个/home/xjia/opt/module/hadoop3.2.1/lib/native,请问是这是正确的吗?




------------------&nbsp;原始邮件&nbsp;------------------
发件人: "zhisheng"<[hidden email]&gt;;
发送时间: 2021年1月5日(星期二) 晚上7:13
收件人: "user-zh"<[hidden email]&gt;;
主题: Re: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist



hi

可以检查一下提交任务的 flink 客户端的 lib 目录下面是否有 flink-sql-parquet_2.11-1.12.0.jar 依赖

Best
zhisheng

冯嘉伟 <[hidden email]&gt; 于2021年1月4日周一 上午9:58写道:

&gt; hi!
&gt;
&gt; java.io.FileNotFoundException: File file:/home/xjia/.flink/...
&gt; 可以看出,从本地加载jar包,而不是hdfs。
&gt;
&gt; 我觉得可能是hadoop环境的问题,导致读取的scheme是file,使用 echo $HADOOP_CLASSPATH 检查你的环境。
&gt;
&gt; Important Make sure that the HADOOP_CLASSPATH environment variable is set
&gt; up
&gt; (it can be checked by running echo $HADOOP_CLASSPATH). If not, set it up
&gt; using
&gt;
&gt;
&gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html
&gt; <
&gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html&gt;;
&gt;
&gt;
&gt;
&gt;
&gt; --
&gt; Sent from: http://apache-flink.147419.n8.nabble.com/
&gt;
Reply | Threaded
Open this post in threaded view
|

回复: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

刘海
In reply to this post by 徐州州
这是我的配置  /etc/profile中的




export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar


export ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper
export HADOOP_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop
export YARN_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn
export HIVE_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive
export HBASE_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase
export SPARK_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark
export OOZIE_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie
export SQOOP_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/sqoop
export KAFKA_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/kafka
export IMPALA_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/impala
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop
export HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs
export HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin


export HADOOP_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"


export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$YARN_HOME/bin:HADOOP_MAPRED_HOME
export PATH=$PATH:$HBASE_HOME/bin:$HIVE_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin
export PATH=$PATH:$OOZIE_HOME/bin:$IMPALA_HOME/bin:$IMPALA_HOME/sbin:$SQOOP_HOME/bin:$KAFKA_HOME/bin


export  PHOENIX_HOME=/opt/apache-phoenix-5.0.0-HBase-2.0-bin/
export PHOENIX_CLASSPATH=$PHOENIX_HOME
export PATH=$PATH:$PHOENIX_HOME/bin


#flink
export YARN_CONF_DIR=/etc/hadoop/conf
export FLINK_HOME=/opt/flink-1.12.0
export PATH=$PATH:$FLINK_HOME/bin
export HADOOP_CLASSPATH=`hadoop classpath`




| |
刘海
|
|
[hidden email]
|
签名由网易邮箱大师定制
在2021年1月6日 09:32,徐州州<[hidden email]> 写道:
我也错在这种问题-(通过crontab启动flink-job失败),使用azkaban调度完美解决。



------------------&nbsp;原始邮件&nbsp;------------------
发件人: "zhisheng"<[hidden email]&gt;;
发送时间: 2021年1月5日(星期二) 晚上7:13
收件人: "user-zh"<[hidden email]&gt;;
主题: Re: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist



hi

可以检查一下提交任务的 flink 客户端的 lib 目录下面是否有 flink-sql-parquet_2.11-1.12.0.jar 依赖

Best
zhisheng

冯嘉伟 <[hidden email]&gt; 于2021年1月4日周一 上午9:58写道:

&gt; hi!
&gt;
&gt; java.io.FileNotFoundException: File file:/home/xjia/.flink/...
&gt; 可以看出,从本地加载jar包,而不是hdfs。
&gt;
&gt; 我觉得可能是hadoop环境的问题,导致读取的scheme是file,使用 echo $HADOOP_CLASSPATH 检查你的环境。
&gt;
&gt; Important Make sure that the HADOOP_CLASSPATH environment variable is set
&gt; up
&gt; (it can be checked by running echo $HADOOP_CLASSPATH). If not, set it up
&gt; using
&gt;
&gt;
&gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html
&gt; <
&gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html&gt;;
&gt;
&gt;
&gt;
&gt;
&gt; --
&gt; Sent from: http://apache-flink.147419.n8.nabble.com/
&gt;
Reply | Threaded
Open this post in threaded view
|

回复: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist

徐州州
thank you


------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "user-zh"                                                                                    <[hidden email]&gt;;
发送时间:&nbsp;2021年1月6日(星期三) 上午9:52
收件人:&nbsp;"[hidden email]"<[hidden email]&gt;;

主题:&nbsp;回复: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist



这是我的配置&nbsp; /etc/profile中的




export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar


export ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper
export HADOOP_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop
export YARN_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn
export HIVE_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive
export HBASE_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase
export SPARK_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark
export OOZIE_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie
export SQOOP_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/sqoop
export KAFKA_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/kafka
export IMPALA_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/impala
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop
export HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs
export HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin


export HADOOP_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"


export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$YARN_HOME/bin:HADOOP_MAPRED_HOME
export PATH=$PATH:$HBASE_HOME/bin:$HIVE_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin
export PATH=$PATH:$OOZIE_HOME/bin:$IMPALA_HOME/bin:$IMPALA_HOME/sbin:$SQOOP_HOME/bin:$KAFKA_HOME/bin


export&nbsp; PHOENIX_HOME=/opt/apache-phoenix-5.0.0-HBase-2.0-bin/
export PHOENIX_CLASSPATH=$PHOENIX_HOME
export PATH=$PATH:$PHOENIX_HOME/bin


#flink
export YARN_CONF_DIR=/etc/hadoop/conf
export FLINK_HOME=/opt/flink-1.12.0
export PATH=$PATH:$FLINK_HOME/bin
export HADOOP_CLASSPATH=`hadoop classpath`




| |
刘海
|
|
[hidden email]
|
签名由网易邮箱大师定制
在2021年1月6日 09:32,徐州州<[hidden email]&gt; 写道:
我也错在这种问题-(通过crontab启动flink-job失败),使用azkaban调度完美解决。



------------------&amp;nbsp;原始邮件&amp;nbsp;------------------
发件人: "zhisheng"<[hidden email]&amp;gt;;
发送时间: 2021年1月5日(星期二) 晚上7:13
收件人: "user-zh"<[hidden email]&amp;gt;;
主题: Re: crontab通过脚本启动flink-job失败,flink-sql-parquet_2.11-1.12.0.jar does not exist



hi

可以检查一下提交任务的 flink 客户端的 lib 目录下面是否有 flink-sql-parquet_2.11-1.12.0.jar 依赖

Best
zhisheng

冯嘉伟 <[hidden email]&amp;gt; 于2021年1月4日周一 上午9:58写道:

&amp;gt; hi!
&amp;gt;
&amp;gt; java.io.FileNotFoundException: File file:/home/xjia/.flink/...
&amp;gt; 可以看出,从本地加载jar包,而不是hdfs。
&amp;gt;
&amp;gt; 我觉得可能是hadoop环境的问题,导致读取的scheme是file,使用 echo $HADOOP_CLASSPATH 检查你的环境。
&amp;gt;
&amp;gt; Important Make sure that the HADOOP_CLASSPATH environment variable is set
&amp;gt; up
&amp;gt; (it can be checked by running echo $HADOOP_CLASSPATH). If not, set it up
&amp;gt; using
&amp;gt;
&amp;gt;
&amp;gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html
&amp;gt; <
&amp;gt; https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/yarn.html&amp;gt;;
&amp;gt;
&amp;gt;
&amp;gt;
&amp;gt;
&amp;gt; --
&amp;gt; Sent from: http://apache-flink.147419.n8.nabble.com/
&amp;gt;