Flink-SQL on yarn 的bug

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink-SQL on yarn 的bug

guaishushu1103@163.com
insert into t_report_realtime_fangxin2_order1

SELECT date_format(TUMBLE_END(w_ts, INTERVAL '60' SECOND),'yyyy-MM-dd') as summary_date,
date_format(TUMBLE_END(w_ts, INTERVAL '60' SECOND), 'yyyy-MM-dd hh') as summary_hour,
date_format(TUMBLE_END(w_ts, INTERVAL '60' SECOND), 'yyyy-MM-dd hh:mm') as summary_minute,
'all' as city1_id,
'all' as cate3_id,
count(DISTINCT(pay_order_id)) as order_num_dj3
FROM (
select w_ts,
JsonIndexOf(ArrayIndexOf(w_data, 0), 'city_id') as city1_id,
JsonIndexOf(ArrayIndexOf(w_data, 0), 'three_level_cate_id') as cate3_id,
JsonIndexOf(ArrayIndexOf(w_data, 0), 'pay_order_id') as pay_order_id
from hdp_lbg_huangye_payorder_binlog
)
GROUP BY TUMBLE(w_ts, INTERVAL '60' SECOND)
;

定义了eventime属性,但是算子并没有watermarks  导致数据一致不能输出


Reply | Threaded
Open this post in threaded view
|

Re: Flink-SQL on yarn 的bug

zhisheng
这个应该不是 bug,如果用代码写,在定义了事件时间的时候,也是要加水印的,否则无法触发窗口的 trigger

[hidden email] <[hidden email]> 于2020年5月15日周五 下午5:36写道:

> insert into t_report_realtime_fangxin2_order1
>
> SELECT date_format(TUMBLE_END(w_ts, INTERVAL '60' SECOND),'yyyy-MM-dd') as
> summary_date,
> date_format(TUMBLE_END(w_ts, INTERVAL '60' SECOND), 'yyyy-MM-dd hh') as
> summary_hour,
> date_format(TUMBLE_END(w_ts, INTERVAL '60' SECOND), 'yyyy-MM-dd hh:mm') as
> summary_minute,
> 'all' as city1_id,
> 'all' as cate3_id,
> count(DISTINCT(pay_order_id)) as order_num_dj3
> FROM (
> select w_ts,
> JsonIndexOf(ArrayIndexOf(w_data, 0), 'city_id') as city1_id,
> JsonIndexOf(ArrayIndexOf(w_data, 0), 'three_level_cate_id') as cate3_id,
> JsonIndexOf(ArrayIndexOf(w_data, 0), 'pay_order_id') as pay_order_id
> from hdp_lbg_huangye_payorder_binlog
> )
> GROUP BY TUMBLE(w_ts, INTERVAL '60' SECOND)
> ;
>
> 定义了eventime属性,但是算子并没有watermarks  导致数据一致不能输出
>
> ------------------------------
> [hidden email]
>