使用mysql-cdc 的scan.startup.mode = specific-offset的读取模式,运行一段时间后,报

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

使用mysql-cdc 的scan.startup.mode = specific-offset的读取模式,运行一段时间后,报

张海深
Hi all:
最近用mysql-cdc的方式,使用Flink-sql整合数据,table的部分配置如下
'debezium.min.row.count.to.stream.results'='1000',
'scan.startup.mode'='specific-offset',
'scan.startup.specific-offset.file'='mybinlog.000029',
'scan.startup.specific-offset.pos'='542607677'
在上述稳定运行接分钟之后,异常抛出了以下错误:
org.apache.kafka.connect.errors.ConnectException: The connector is trying to read binlog starting at GTIDs 36b8bd0f-a435-11eb-9962-b4055d9ecb74:21820662-21926801,d713fc3c-afc6-11eb-a7b8-b4055d9ec5e6:185910-192349 and binlog file 'mybinlog.000029', pos=808323397, skipping 6 events plus 1 rows, but this is no longer available on the server. Reconfigure the connector to use a snapshot when needed.
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:133)
at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:106)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:758)
at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:171)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:834)
请问是什么情况下导致这个问题。
Reply | Threaded
Open this post in threaded view
|

Re:使用mysql-cdc 的scan.startup.mode = specific-offset的读取模式,运行一段时间后,报

董建



我也遇到了这种情况,可能是你们的db做了主从切换。
因为binlog每台服务器的pos都不一样。
mysql5.6以后支持了GTID的同步方式,这个是全局唯一的。但是目前mysql-cdc貌似还不支持。
我目前的解决方案是出错后从最后的位置开始消费,可能会丢失一部分数据。














在 2021-06-07 16:00:21,"张海深" <[hidden email]> 写道:

>Hi all:
>最近用mysql-cdc的方式,使用Flink-sql整合数据,table的部分配置如下
>'debezium.min.row.count.to.stream.results'='1000',
>'scan.startup.mode'='specific-offset',
>'scan.startup.specific-offset.file'='mybinlog.000029',
>'scan.startup.specific-offset.pos'='542607677'
>在上述稳定运行接分钟之后,异常抛出了以下错误:
>org.apache.kafka.connect.errors.ConnectException: The connector is trying to read binlog starting at GTIDs 36b8bd0f-a435-11eb-9962-b4055d9ecb74:21820662-21926801,d713fc3c-afc6-11eb-a7b8-b4055d9ec5e6:185910-192349 and binlog file 'mybinlog.000029', pos=808323397, skipping 6 events plus 1 rows, but this is no longer available on the server. Reconfigure the connector to use a snapshot when needed.
>at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:133)
>at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:106)
>at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:758)
>at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:171)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
>at java.lang.Thread.run(Thread.java:834)
>请问是什么情况下导致这个问题。
Reply | Threaded
Open this post in threaded view
|

Re:Re:使用mysql-cdc 的scan.startup.mode = specific-offset的读取模式,运行一段时间后,报

casel.chen
这个问题很严重啊,生产线上可不敢这么用,丢失部分数据是不能接受的。社区什么时候能支持 GTID 呢?官方网档上有写么?

















在 2021-06-07 18:40:50,"董建" <[hidden email]> 写道:

>
>
>
>我也遇到了这种情况,可能是你们的db做了主从切换。
>因为binlog每台服务器的pos都不一样。
>mysql5.6以后支持了GTID的同步方式,这个是全局唯一的。但是目前mysql-cdc貌似还不支持。
>我目前的解决方案是出错后从最后的位置开始消费,可能会丢失一部分数据。
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>在 2021-06-07 16:00:21,"张海深" <[hidden email]> 写道:
>>Hi all:
>>最近用mysql-cdc的方式,使用Flink-sql整合数据,table的部分配置如下
>>'debezium.min.row.count.to.stream.results'='1000',
>>'scan.startup.mode'='specific-offset',
>>'scan.startup.specific-offset.file'='mybinlog.000029',
>>'scan.startup.specific-offset.pos'='542607677'
>>在上述稳定运行接分钟之后,异常抛出了以下错误:
>>org.apache.kafka.connect.errors.ConnectException: The connector is trying to read binlog starting at GTIDs 36b8bd0f-a435-11eb-9962-b4055d9ecb74:21820662-21926801,d713fc3c-afc6-11eb-a7b8-b4055d9ec5e6:185910-192349 and binlog file 'mybinlog.000029', pos=808323397, skipping 6 events plus 1 rows, but this is no longer available on the server. Reconfigure the connector to use a snapshot when needed.
>>at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:133)
>>at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:106)
>>at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:758)
>>at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:171)
>>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
>>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
>>at java.lang.Thread.run(Thread.java:834)
>>请问是什么情况下导致这个问题。