v4.11.3

DatabaseChanges mode improvements

• Fixed Clickhouse table fetching to ignore materialized columns when inserting data. If a column is materialized from another one, the column is skipped for data insertion coming from the Substreams.

• Added handle_block_duration to postgres sink stats periodic logger to show how long from start to finish handling a block takes (this make more sense once live where each block is flushed to the database normally).

• Added new Prometheus metrics:

  • substreams_sink_sql_prune_reversible_segment_duration - How long it takes to prune the reversible of the chain, when there is one
  • substreams_sink_sql_tx_query_execution_duration{query_type: normal|undo} - How long it took to execute the tx.ExecContext call for saving the received data into the database, normal being for the data insertion, undo being for the saving of "undo" row to handle re-orgs.

  Note Current metrics starts with substreams_sink_postgres, we have started to now use substreams_sink_sql.

v4.11.2

DatabaseChanges mode improvements

• Fixed Clickhouse table fetching to ignore materialized views, normal views as well as temporary tables.

v4.11.1

Clickhouse from proto improvements

Add retry logic queries and enhance client handling

• Introduced clickhouse-query-retry-count and clickhouse-query-retry-sleep flags to configure retries.
• Implemented retry mechanism with fresh client acquisition for query execution failures.
• Enhanced ClickHouse database and inserter logic to handle retries consistently.

DatabaseChanges mode improvements

• Refactor encoder to use CSV writer for row formatting when using substreams-sink-sql generate-csv command.

Clickhouse specific changes

• Breaking: ClickHouse HTTP protocol is no longer supported. Only the native TCP protocol is accepted (ports 9000 or 9440 for secure connections). Attempting to use HTTP ports (8123 or 8443) will result in an explicit error: ClickHouse HTTP protocol (port <port>) is not supported. Please use the native TCP protocol on port 9000 or 9440. The HTTP protocol had compatibility issues with schema introspection and metadata queries.

v4.11.0

• Added support for nested objects in postgres from-proto mode. Table with nested objects will be created with JSONB type.

v4.10.0

• Added support for one level deep nested objects in clickhouse from-proto mode. Table with nested objects will be created with Nested type.
• Fix postgres handling of type bytea[]

v4.9.0

DatabaseChanges mode improvements

• Fixed bug where some unescape column where not correctly preserving cases in some situations like handling of upsert primary key(s).

From-proto mode improvements

• Fixed conversion of Decimal128, Decimal256, Int128, UInt128, Int256, and UInt256 if they are defined as optional in the Protobuf.

Clickhouse specific improvements

• Removed ClickhouseReplacingField and associated functionality
  • Removed ClickhouseReplacingField usage from protobuf schema and generated files
  • Adjusted ClickhouseTableOptions to exclude replacing_fields
  • Updated ClickHouse dialect to streamline ReplacingMergeTree logic and add allow_experimental_replacing_merge_with_cleanup setting
  • Cleaned up dependencies and unused code related to ClickhouseReplacingField

v4.8.0

From-proto mode enhancements

• Added support for new numeric types in from-proto mode: Decimal128, Decimal256, Int128, UInt128, Int256, and UInt256
  • These fields are converted from string format in both PostgreSQL and ClickHouse implementations
  • PostgreSQL maps these types to appropriate NUMERIC and DECIMAL data types with proper precision and scale
  • ClickHouse maps these types to their corresponding native numeric types (Int128, UInt128, Int256, UInt256, Decimal128, Decimal256)
• Enhanced protobuf schema definitions to include string conversion types for large numeric values
• Added string-to-numeric conversion support in PostgreSQL type mapping system

v4.7.0

• Added support for optional fields in from-proto mode.
• Added support for repeated native fields in from-proto mode.
• Added bytes encoding support. use flags --bytes-encoding with values raw, hex, 0xhex, base64, base58

v4.6.9

• Bumped substreams to v1.16.7-0 to add foundational stores support.

v4.6.8

Postgres & substreams-sink-sql run

• Fixed schemaName=<schema> not working properly on setup command if the user's SQL script does not specify an specific schema itself.

Clickhouse & substreams-sink-sql run

• Skip AggregateFunction columns on Clickhouse when describing the schema.