Handling of large data

[oduerr]

Handling of large data. It should be possible to load data from repositories like dropbox. They should be CSV files or gz.csv.

[oduerr]

Description

We aim to create a flexible and platform-independent caching system for CSV files, supporting both R and Python. This system will enable easy caching, loading, and documentation of datasets. The main components for each dataset are:

• CSV File: Contains the actual data [Optional]
• YAML File: Stores metadata about the dataset and can provide external sources for large data [Optional]
• Markdown File: Provides additional documentation and visualization related to the dataset [Optional]

Tasks

Data Management and Caching:

• Implement a flexible load function to support both local and external datasets.
• Load CSV files, convert them to DataFrames, and cache them.
• Use YAML files to store metadata like source and documentation, and cache

Separation of Large and Small Files:

• Store small datasets directly in the repository.
• Store large datasets externally (e.g., Zenodo) and reference them in YAML files.

Documentation:

• Render Quarto files in Markdown to provide documentation and example plots.
• Ensure documentation can be used without the package itself.