flink sql cli 读取 hbase表报错

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

flink sql cli 读取 hbase表报错

王良
您好:

             我使用的是flink 1.10 ,通过sql-client 创建了hbase 表

CREATE TABLE dim_term (
    term_id string,
    info ROW(
    term_name string,
    term_name_combine string,
    term_notice string,
    term_remarks string,
    season string,
    term_sequence string,
    term_start_time string,
    term_end_time string,
    term_description string,
    term_status int,
    is_mvp_term int,
    ctime string,
    utime string
    )

) WITH (
'connector.type' = 'hbase',
'connector.version' = '1.4.3',
'connector.table-name' = 'dim_term',
'connector.zookeeper.quorum' = 'emr-header-1.cluster-109533:2181,emr-worker-1.cluster-109533:2181,emr-header-2.cluster-109533:2181',
'connector.zookeeper.znode.parent' = '/hbase'
)

遇到的问题是,当我在sql-client ,执行select * from dim_term 的时候报错

2020-06-29 11:26:51,718 INFO  org.apache.flink.addons.hbase.HBaseRowInputFormat             org.apache.flink.addons.hbase.HBaseRowInputFormat.configure(HBaseRowInputFormat.java:65) - Initializing HBase configuration.
2020-06-29 11:26:51,831 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:120) - Process identifier=hconnection-0x57b9485d connecting to ZooKeeper ensemble=localhost:2181


org.apache.flink.addons.hbase.HBaseRowInputFormat,这个类里面没有读取到zookeeper的配置
 
Reply | Threaded
Open this post in threaded view
|

Re: flink sql cli 读取 hbase表报错

Leonard Xu
Hello,
这应该是一个已知bug[1],原因是Configuration是不可序列化的,HbaseRowInputFormat中没有正确处理,导致用户DDL中的zk配置无法传递。
在flink1.11和1.12上已经修复。如果是1.10.x版本中,可以将HBase 的配置文件(hbase-default.xml、 hbase-site.xml) 添加到 classpath下,也可以把
HBase 的配置文件添加到HADOOP_CLASSPATH(flnk启动脚本会检查HADOOP_CLASSPATH环境变量并加载),两种方式Flink集群和SQL Client都能加载到Hbase的配置文件,从而加载到配置文件中正确的zk信息。


祝好,
Leonard
[1]  https://issues.apache.org/jira/browse/FLINK-17968


> 在 2020年6月29日,11:45,王良 <[hidden email]> 写道:
>
> 您好:
>
>             我使用的是flink 1.10 ,通过sql-client 创建了hbase 表
>
> CREATE TABLE dim_term (
>    term_id string,
>    info ROW(
>    term_name string,
>    term_name_combine string,
>    term_notice string,
>    term_remarks string,
>    season string,
>    term_sequence string,
>    term_start_time string,
>    term_end_time string,
>    term_description string,
>    term_status int,
>    is_mvp_term int,
>    ctime string,
>    utime string
>    )
>
> ) WITH (
> 'connector.type' = 'hbase',
> 'connector.version' = '1.4.3',
> 'connector.table-name' = 'dim_term',
> 'connector.zookeeper.quorum' = 'emr-header-1.cluster-109533:2181,emr-worker-1.cluster-109533:2181,emr-header-2.cluster-109533:2181',
> 'connector.zookeeper.znode.parent' = '/hbase'
> )
>
> 遇到的问题是,当我在sql-client ,执行select * from dim_term 的时候报错
>
> 2020-06-29 11:26:51,718 INFO  org.apache.flink.addons.hbase.HBaseRowInputFormat             org.apache.flink.addons.hbase.HBaseRowInputFormat.configure(HBaseRowInputFormat.java:65) - Initializing HBase configuration.
> 2020-06-29 11:26:51,831 INFO  org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper        org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:120) - Process identifier=hconnection-0x57b9485d connecting to ZooKeeper ensemble=localhost:2181
>
>
> org.apache.flink.addons.hbase.HBaseRowInputFormat,这个类里面没有读取到zookeeper的配置