flink1.11 sql 发布到yarn session时找不到hbase相关的类

[junbaozhang]

Hi,all：
最近在升级flink1.11，sql中用到hbase connctor，发布到yarn-session时，报如下异常：
2020-07-29 11:49:55
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer
at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:221)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer
at org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:1053)
at org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:496)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
... 7 more

注意到官方文档有Note:
Note: To use HBase connector in SQL Client or Flink cluster, it’s highly recommended to add HBase dependency jars to Hadoop classpath. Flink will load all jars under Hadoop classpath automatically, please refer to HBase, MapReduce, and the CLASSPATH<https://hbase.apache.org/book.html#hbase.mapreduce.classpath> about how to add HBase dependency jars to Hadoop classpath.
          但是在yarn session下怎么设置classpath呢

[Xintong Song]

export HADOOP_CLASSPATH 就可以了


Thank you~

Xintong Song



On Wed, Jul 29, 2020 at 6:43 PM [hidden email] <
[hidden email]> wrote:

> Hi,all：
> 最近在升级flink1.11，sql中用到hbase connctor，发布到yarn-session时，报如下异常：
> 2020-07-29 11:49:55
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.util.ByteStringer
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:221)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.util.ByteStringer
> at
> org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:1053)
> at
> org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:496)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> ... 7 more
>
> 注意到官方文档有Note:
> Note: To use HBase connector in SQL Client or Flink cluster, it’s highly
> recommended to add HBase dependency jars to Hadoop classpath. Flink will
> load all jars under Hadoop classpath automatically, please refer to HBase,
> MapReduce, and the CLASSPATH<
> https://hbase.apache.org/book.html#hbase.mapreduce.classpath> about how
> to add HBase dependency jars to Hadoop classpath.
>           但是在yarn session下怎么设置classpath呢
>

[junbaozhang]

Hi，Xintong：
是把相关的jar配到hadoop_classpath然后再export吗？

Best,
Junbao Zhang
________________________________
发件人: Xintong Song <[hidden email]>
发送时间: 2020年7月30日 10:28
收件人: user-zh <[hidden email]>
主题: Re: flink1.11 sql 发布到yarn session时找不到hbase相关的类

export HADOOP_CLASSPATH 就可以了


Thank you~

Xintong Song



On Wed, Jul 29, 2020 at 6:43 PM [hidden email] <
[hidden email]> wrote:

> Hi,all：
> 最近在升级flink1.11，sql中用到hbase connctor，发布到yarn-session时，报如下异常：
> 2020-07-29 11:49:55
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.util.ByteStringer
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:248)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:221)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.util.ByteStringer
> at
> org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:1053)
> at
> org.apache.hadoop.hbase.protobuf.RequestConverter.buildScanRequest(RequestConverter.java:496)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> ... 7 more
>
> 注意到官方文档有Note:
> Note: To use HBase connector in SQL Client or Flink cluster, it’s highly
> recommended to add HBase dependency jars to Hadoop classpath. Flink will
> load all jars under Hadoop classpath automatically, please refer to HBase,
> MapReduce, and the CLASSPATH<
> https://hbase.apache.org/book.html#hbase.mapreduce.classpath> about how
> to add HBase dependency jars to Hadoop classpath.
>           但是在yarn session下怎么设置classpath呢
>