BYOL notebooks assume local dataset paths without download or structure instructions.

[nikhilchhokar]

Hi ML4Sci team,

While continuing to run the BYOL self-supervised learning notebooks
(model1_byol.ipynb), I encountered a FileNotFoundError when loading data:

The notebook assumes local directories such as:

• Model_I/no_sub
• Model_I/axion
• Model_I/cdm

However, there are no instructions on:

• how to obtain the dataset
• expected directory structure
• where these folders should be placed

As a result, the notebook cannot be run end-to-end on a fresh setup without
guessing the dataset layout.

I wanted to confirm the intended dataset source and structure before attempting
any fixes or documentation updates.

Thanks!

[yash-755]

Hi ML4Sci team,

I’m a new contributor and I can take this up if that’s okay.
I’ve been able to reproduce the issue on a fresh setup while running the BYOL notebooks.

Before making changes, I wanted to confirm the intended dataset source and
expected directory structure so the documentation updates are accurate.

Happy to work on documenting the dataset and folder layout once confirmed.

Thanks!

[nikhilchhokar]

Thanks for offering to help!
I’ve also been able to reproduce the dataset issue with the BYOL notebooks — they currently assume local folders like Model_I/no_sub, Model_I/axion, and Model_I/cdm without instructions on where to get the data or how to organize it.
My current focus has been on documenting the setup/dependencies so the notebooks can run end-to-end on a fresh environment (#84).
If you have clarity on the intended dataset source and directory layout, that would be perfect — I can then include accurate instructions in the documentation and avoid overlap. Happy to coordinate!

[yash-755]

Thanks for the clarification.

I searched through model1_byol.ipynb and found that the notebooks
explicitly reference a local Model_I directory in multiple places
(e.g. Model_I/axion, Model_I/cdm, Model_I/no_sub).

From what I can see, the code assumes this folder structure already
exists locally, but the README doesn’t explain where the data comes
from or how the Model_I directory should be created.

I can help by clearly documenting the expected folder structure
(and noting that the dataset source is not currently specified),
so new users understand what is required to run the notebooks.

[shubhika123]

Hi everyone,

I looked through model1_byol.ipynb and noticed that the data paths are directly referenced in the notebook itself, with several cells assuming the presence of a local Model_I/ directory (for example: Model_I/axion, Model_I/cdm, and Model_I/no_sub).

From what I can tell, the repository currently doesn’t mention:

• where this dataset is expected to come from,
• whether Model_I should be downloaded, generated, or manually created,
• or what the intended directory layout is for running the notebooks.

Before attempting any documentation updates or a PR, it might be helpful to clarify a few points:

1. Is Model_I based on a specific public DeepLense dataset?
2. Is there a standard folder structure that users are expected to follow?
3. Should the notebooks handle missing data more explicitly, or is manual setup expected?

Once this is clarified, I’d be happy to help with documenting the dataset requirements or setup steps so the notebooks can run smoothly on a fresh setup.

[yash-755]

Thanks for looking into this.

I’m seeing the same assumptions around the local Model_I directory
across the BYOL notebooks. Happy to help document the expected folder
structure once the dataset source and intended setup are clarified.

[nikhilchhokar]

Thanks for confirming, I’m seeing the same behavior while running the BYOL notebooks locally.
I’m also exploring this as a contributor and haven’t heard back from maintainers yet.
Agree that it makes sense to wait for clarification on the dataset source and expected directory structure before making any changes or PRs.

[aazeeem11]

Hi @nikhilchhokar @yash-755,

I've looked into this and I believe I have identified the missing data source. The Model_I directory referenced in the notebook corresponds to the standard DeepLense Model 1 (Supervised Learning) dataset, which contains the axion, cdm, and no_sub classes.

This dataset is historically used in ML4SCI tasks but isn't included in the repo to keep the size down. The notebook currently assumes the user has manually placed the folder there, which causes the FileNotFoundError.

Proposed Solution:
I can submit a PR that adds a setup cell to automatically check for the data and download/extract it if it's missing. This ensures the notebook runs end-to-end on Colab or a fresh local setup without manual intervention.

Here is the implementation I am proposing:

import os
import tarfile
import urllib.request

# Configuration
DATASET_URL = "https://github.com/ML4SCI/ML4SCI_GSoC/raw/main/DeepLense/Domain%20Adaptation/lenses.tgz"
DATA_DIR = "Model_I"

def setup_data():
    if os.path.exists(DATA_DIR):
        print(f"Dataset '{DATA_DIR}' already exists.")
        return

    print(f"Dataset not found. Downloading from {DATASET_URL}...")
    filename = "lenses.tgz"
    try:
        urllib.request.urlretrieve(DATASET_URL, filename)
        print("Download complete. Extracting...")
        
        with tarfile.open(filename, "r:gz") as tar:
            tar.extractall()
        print("Extraction complete.")
        
        # Cleanup
        if os.path.exists(filename):
            os.remove(filename)
            
    except Exception as e:
        print(f"Error during setup: {e}")

# Run setup
setup_data()

Shall I open a PR to integrate this fix into the notebook?

[VipS-2004]

Hi, I’d like to work on this issue and help improve the dataset setup instructions for the BYOL notebooks.
Please let me know if I can take this up.

[Apprentice2907]

Hello

I’ve added a fix to address the missing dataset setup in the BYOL notebook.

The issue was that model1_byol.ipynb assumes the presence of the Model_I/ directory without documenting the dataset source or setup steps, which causes a FileNotFoundError on a fresh setup.

To resolve this, I:

added a small setup script to download and extract the required Model_I dataset from the DeepLenseSim repository,

added clear dataset setup instructions and a safety check at the top of the BYOL notebook,

updated the local README with explicit setup steps.

This allows the notebook to run end-to-end on a clean environment without guessing the dataset structure.

I’ve opened a PR with these changes and would appreciate your feedback.
Thanks!

[VipS-2004]

Thanks for the update!
My PR focuses on improving README clarity and onboarding for first-time contributors, while your changes address the dataset setup directly in the notebook.
Happy to coordinate or adjust if needed.