ETL-516: Update to pipeline v2

[PabloPardoGarcia]

No description provided.

[github-actions]

Test Coverage Report

File	Stmts	Miss	Cover	Missing
glassflow/etl
__init__.py	5	0	100%
api_client.py	64	5	92%	70–73, 123
client.py	39	4	89%	59–61, 141
dlq.py	41	2	95%	45, 70
errors.py	27	0	100%
pipeline.py	146	7	95%	334–337, 350, 406–407
tracking.py	26	0	100%
utils.py	11	11	0%	1, 3, 6, 8–11, 17–18, 22, 28
glassflow/etl/models
__init__.py	10	0	100%
base.py	10	4	60%	7–10
config.py	34	8	76%	37, 40–41, 43–45, 47–48
data_types.py	77	0	100%
filter.py	17	8	52%	12, 15–18, 20–22
join.py	51	2	96%	44, 80
metadata.py	4	0	100%
pipeline.py	89	3	96%	135, 145, 148
schema.py	26	0	100%
sink.py	61	7	88%	33, 39, 47, 49, 53, 55, 61
source.py	132	7	94%	47, 51, 68, 81, 124, 150, 166
TOTAL	870	68	92%

[github-actions]

Test Coverage Report

File	Stmts	Miss	Cover	Missing
etl
__init__.py	5	0	100%
api_client.py	64	5	92%	70–73, 123
client.py	39	4	89%	59–61, 141
dlq.py	41	2	95%	45, 70
errors.py	27	0	100%
pipeline.py	146	7	95%	334–337, 350, 406–407
tracking.py	26	0	100%
utils.py	11	11	0%	1, 3, 6, 8–11, 17–18, 22, 28
etl/models
__init__.py	10	0	100%
base.py	10	4	60%	7–10
config.py	34	8	76%	37, 40–41, 43–45, 47–48
data_types.py	77	0	100%
filter.py	17	8	52%	12, 15–18, 20–22
join.py	51	2	96%	44, 80
metadata.py	4	0	100%
pipeline.py	89	3	96%	135, 145, 148
schema.py	26	0	100%
sink.py	61	7	88%	33, 39, 47, 49, 53, 55, 61
source.py	132	7	94%	47, 51, 68, 81, 124, 150, 166
TOTAL	870	68	92%

[ashish-bagri]

doesn't the migration script need to handle the case with dedup_key not in mapping ?

[PabloPardoGarcia]

The script will first add all the fields in in the topic schemas with source_id, name and type. Then will add the column_name and column_type to the fields that are mapped to the ClickHouse table in the table_mapping.

So it will handle those cases. It will also add other fields that are not mapped but appear in the topic schema