flink1.10 sql state 聚合函数 窗口统计,状态越来越大,无法自动清理问题

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

flink1.10 sql state 聚合函数 窗口统计,状态越来越大,无法自动清理问题

zhuyuping
<http://apache-flink.147419.n8.nabble.com/file/t912/e1111e0a-24c3-4e40-b830-1eec9f183a02.png>  

现在flink 使用
如下sql:
我创建了一个聚合函数,就是mapview 简单的put ,然后返回string

然后我使用翻转窗口10s ,1s,1分钟进行统计,
但是出现了每隔3个 ,状态就会增大。最后状态会越来越大,导致checkpoint失败,任务重启,
刚开始以为是反压。最后我使用insert into discardSink ,也是出现同样的问题
sql:
CREATE VIEW cpd_xx_view as select
imei,incrmentFeatureCollect(CAST(serverTime AS INT),adId) as feature from
xxx_view
  group by TUMBLE (proctime, INTERVAL '10' SECOND),imei;  






--
Sent from: http://apache-flink.147419.n8.nabble.com/