Flink-1.11 消费Kafka 写 Hive,消费堆积问题

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Flink-1.11 消费Kafka 写 Hive,消费堆积问题

nashcen
*flink 消费kafka平均速度 = (8810108 - 8646583)/5  = 32705条/分钟*
每小时大约消费 200w 条数据,速度太慢了,目前已经堆积了800w条数据,这种情况该如何处理?

*21/01/27 20:05:46* INFO clients.Metadata: Cluster ID:
h-faeyjNRhS5xcAUy1JR2Q
Consumer group 'stg_dcpoints_hive' has no active members.

TOPIC             PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG            
CONSUMER-ID     HOST            CLIENT-ID
ods_dcpoints_prod 0          6599508940      6608319048      *8810108 *      
-               -               -

*21/01/27 20:10:45* INFO clients.Metadata: Cluster ID:
h-faeyjNRhS5xcAUy1JR2Q
Consumer group 'stg_dcpoints_hive' has no active members.

TOPIC             PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG            
CONSUMER-ID     HOST            CLIENT-ID
ods_dcpoints_prod 0          6599672465      6608319048      *8646583 *      
-               -               -



--
Sent from: http://apache-flink.147419.n8.nabble.com/