[Bug] flink sqlserver cdc After full completion and multiple checkpoints, the increment was not executed [Concurrency is 2] #2376
Description
Search before asking
- I searched in the issues and found nothing similar.
Flink version
1.17.1
Flink CDC version
2.4.1
Database and its version
�sqlserver 2019-latest
Minimal reproduce step
USE [test];
GO
IF OBJECT_ID('cdc_source_100', 'U') IS NOT NULL
DROP TABLE [cdc_source_100];
GO
CREATE TABLE [cdc_source_100] (
[id] INT IDENTITY(1,1) NOT NULL,
[name] VARCHAR(255) NOT NULL,
[age] INT NOT NULL,
[address] VARCHAR(255) NOT NULL,
[create_time] DATETIME NOT NULL,
[update_time] DATETIME NOT NULL,
CONSTRAINT [PK_cdc_source_100] PRIMARY KEY ([id])
) ON [PRIMARY];
CREATE TABLE cdc_target_100 (
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(255) NOT NULL,
age int(11) NOT NULL,
address varchar(255) NOT NULL,
create_time datetime NOT NULL,
update_time datetime NOT NULL,
insert_time datetime DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
Flink-SQL:
CREATE CATALOG target WITH(
'type' = 'jdbc',
'default-database' = 'test',
'username' = 'root',
'password' = 'yunli2@AUG',
'base-url' = 'jdbc:mysql://172.30.13.182:3306'
);
--100万
-- 创建sqlserver-cdc source
CREATE TABLE cdc_source (
id INT,
name STRING,
age INT,
address STRING,
create_time TIMESTAMP(3),
update_time TIMESTAMP(3),
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'sqlserver-cdc',
'hostname' = '172.30.13.181',
'port' = '11433',
'username' = 'debezium',
'password' = 'xmb@123456',
'database-name' = 'test',
'table-name' = 'dbo.cdc_source_100'
);
-- 执行写入操作
insert into target.test.cdc_target_100 select *,now() as insert_time from cdc_source;
-- 并发度为2, 全量初始化100万数据正常, 如果只是update某一条数据,正常同步, 之后执行操作新增100万数据,就不再同步了,经过多次checkpoint
insert into [cdc_source_100] ([name], [age], [address], [create_time], [update_time]) select Top 1000000 [name], [age], [address], [create_time], [update_time] from [cdc_source];
What did you expect to see?
I hope that after the full completion of sqlserver-cdc, after a checkpoint, the increment can proceed normally
What did you see instead?
After completing full execution, after multiple checkpoints, no incremental steps are taken
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!