Historical duplicate entities

[stwiname]

Background

With historical indexing we record changes in an entity over time/blocks. Currently when you set an entity it will always create a new record in the DB with a unique _id and close the block range of the previous record.

A problem arises when wanting to upsert an entity that already exists but the entity has not changed. This will still follow the same process as if the entity has changed and create a new record even though no fields of the entity have changed.

Goal

Ideally when an entity is upserted but there are no changes to the fields then it should be a noop as the entity hasn't actually changed. The solution to this should be more efficient than querying for an existing entity, comparing it in a handler and deciding whether to update or not. It should also not have a significant performance and resource impact that outweighs the benefits of having duplicate entities.

Example

Current behaviour

• Block N: Upsert entity A. This will create a new db entry with with the fields of entity A, a unique _id field and a _block_range: [n,)
• Block N+M: Upsert entity A' (with the same fields as A). This will create a new db entry with with the fields of entity A', a unique _id field and a _block_range: [n+m,) and will update the _block_range of the record started at N to [n,n+m).

New behaviour

• Block N: Upsert entity A. This will create a new db entry with with the fields of entity A, a unique _id field and a _block_range: [n,)
• Block N+M: Upsert entity A'. Check if the fields of A' are the same as A.
  • If they are equal: Make no DB changes
  • If they are different: This will create a new db entry with with the fields of entity A', a unique _id field and a _block_range: [n+m,) and will update the _block_range of the record started at N to [n,n+m).