Skip to content

In-Memory Datase w/ Zarr embeddings backend #229

@sebastian9991

Description

@sebastian9991

This scalability approach still assumes that we have the full edge_index in memory, however, perhaps we can leverage zarr to read/write the embeddings from disk as we are iterating over our batches. This will keep the memory requirement low as we would not need to allocate space for the feature matrix. Simply read by indices needed per batch.

The question, however, is how we will write the nth-layer embeddins back to the

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions