float-double multiplication error during phase 2 training

[sbgonenc]

Report

Hi Manu!

I hope everything is going well and thanks for the tool. I have came across an issue which might be a bug. After the model finished the training in the first phase, it threw this error during the initialization and after extracting latent topics for the second phase:

Error:

/fast/AG_Bunina/Berk/projects/scDoRI/src/scdori/_core/train_grn.py:143: RuntimeWarning: invalid value encountered in divide
  rna_tf_vals = median_cell * (rna_tf_vals / rna_tf_vals.sum(axis=1, keepdims=True))
INFO:scdori._core.train_grn:Starting GRN training

GRN Epoch 0:   0%|          | 0/2417 [00:00<?, ?it/s]
GRN Epoch 0:   0%|          | 0/2417 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/fast/AG_Bunina/Berk/projects/scDoRI/train_model.py", line 145, in <module>
    model = train_model_grn(
            ^^^^^^^^^^^^^^^^
  File "/fast/AG_Bunina/Berk/projects/scDoRI/src/scdori/_core/train_grn.py", line 465, in train_model_grn
    out = model(
          ^^^^^^
  File "/fast/AG_Bunina/Berk/envs/mamba/envs/scDoRI/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fast/AG_Bunina/Berk/envs/mamba/envs/scDoRI/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fast/AG_Bunina/Berk/projects/scDoRI/src/scdori/_core/models.py", line 339, in forward
    C[topic] = tf_expression_input[topic] @ G_act + tf_expression_input[topic] @ G_rep
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: double != float

I can share the relating data and pre-trained model, when needed.

Version/OS Info:

| Package      | Version             |
| ------------ | ------------------- |
| pandas       | 2.3.1               |
| anndata      | 0.12.1              |
| torch        | 2.8.0 (2.8.0+cu128) |
| numpy        | 1.26.4              |
| scikit-learn | 1.7.1               |
| scdori       | 0.0.1               |

| Dependency        | Version     |
| ----------------- | ----------- |
| statsmodels       | 0.14.5      |
| executing         | 2.2.0       |
| asttokens         | 3.0.0       |
| scipy             | 1.15.3      |
| jupyter_client    | 8.6.3       |
| tangermeme        | 0.3.0       |
| ipython           | 9.4.0       |
| texttable         | 1.7.0       |
| tqdm              | 4.67.1      |
| ipywidgets        | 8.1.7       |
| llvmlite          | 0.44.0      |
| platformdirs      | 4.3.8       |
| traitlets         | 5.14.3      |
| pyarrow           | 14.0.2      |
| six               | 1.17.0      |
| pynndescent       | 0.5.13      |
| legacy-api-wrap   | 1.4.1       |
| jupyter_core      | 5.8.1       |
| matplotlib        | 3.10.5      |
| prompt_toolkit    | 3.0.51      |
| patsy             | 1.0.1       |
| h5py              | 3.14.0      |
| ipykernel         | 6.30.1      |
| packaging         | 25.0        |
| gtfparse          | 2.5.0       |
| triton            | 3.4.0       |
| python-dateutil   | 2.9.0.post0 |
| psutil            | 7.0.0       |
| donfig            | 0.8.1.post1 |
| Pygments          | 2.19.2      |
| comm              | 0.2.3       |
| fsspec            | 2025.7.0    |
| polars            | 0.20.31     |
| pyBigWig          | 0.3.24      |
| threadpoolctl     | 3.6.0       |
| igraph            | 0.11.9      |
| defusedxml        | 0.7.1       |
| kiwisolver        | 1.4.8       |
| mpmath            | 1.3.0       |
| pyfaidx           | 0.8.1.4     |
| pytz              | 2025.2      |
| pyzmq             | 27.0.1      |
| pure_eval         | 0.2.3       |
| scanpy            | 1.11.4      |
| seaborn           | 0.13.2      |
| jedi              | 0.19.2      |
| pillow            | 11.3.0      |
| pyparsing         | 3.2.3       |
| cffi              | 1.17.1      |
| typing_extensions | 4.14.1      |
| joblib            | 1.5.1       |
| zarr              | 3.1.1       |
| umap-learn        | 0.5.9.post2 |
| wcwidth           | 0.2.13      |
| leidenalg         | 0.10.2      |
| stack-data        | 0.6.3       |
| session-info2     | 0.2         |
| crc32c            | 2.7.1       |
| parso             | 0.8.4       |
| decorator         | 5.2.1       |
| pycparser         | 2.22        |
| setuptools        | 80.9.0      |
| natsort           | 8.4.0       |
| PyYAML            | 6.0.2       |
| numba             | 0.61.2      |
| numcodecs         | 0.16.1      |
| sympy             | 1.14.0      |
| tornado           | 6.5.1       |
| cycler            | 0.12.1      |
| debugpy           | 1.8.16      |

| Component | Info                                                                           |
| --------- | ------------------------------------------------------------------------------ |
| Python    | 3.12.11 | packaged by conda-forge | (main, Jun  4 2025, 14:45:31) [GCC 13.3.0] |
| OS        | Linux-5.14.0-427.42.1.el9_4.x86_64-x86_64-with-glibc2.34                       |
| Updated   | 2025-09-03 10:32                                                               |

Thanks again for your interest! And feel free to contact when needed!
Berk

Versions

[DmitriiSeverinov]

Hi Berk,

I had a similar issue, while I was trying to de-bug another issue...
So, at first I got this error after the first phase of training:

INFO:scdori._core.train_grn:Encoder is now frozen in GRN phase.
INFO:scdori._core.train_grn:Peak-gene links are now frozen in GRN phase.
INFO:scdori._core.train_grn:Topic-peak decoder is now frozen in GRN phase.
INFO:scdori._core.train_grn:Topic-tf decoder is now frozen in GRN phase.
Extracting latent topics: 100%|██████████| 678/678 [06:11<00:00,  1.82it/s]
/data/horse/ws/dmse952c-scDoRI_env/scDoRI/src/scdori/_core/train_grn.py:143: RuntimeWarning: invalid value encountered in divide
  rna_tf_vals = median_cell * (rna_tf_vals / rna_tf_vals.sum(axis=1, keepdims=True))

It was leading to RNA and TF loss being nan.
My hypothesis is that it happens, when in a certain topic none of the TF is expressed, because if we look into the code (_core/train_grn.py), we see that if the expression mode == True, we divide the expression of a TF within a topic by sum of the expression of all TFs within that topic.
So, rna_tf_vals.sum(axis=1, keepdims=True) will give you a vector of TF expression sums. If within a topic none of the TFs is expressed, you will encounter 0/0 division in rna_tf_vals / rna_tf_vals.sum(axis=1, keepdims=True), which produces nan values.
To address this issue, I first attempted with this:

    if tf_expression_mode == "True":
        latent_all_torch = get_latent_topics(
            model, device, train_loader, rna_anndata, atac_anndata, num_cells, tf_indices, encoding_batch_onehot
        )
        top_k_indices = np.argsort(latent_all_torch, axis=0)[-config_file.cells_per_topic :]
        rna_tf_vals = rna_anndata.X[:, tf_indices]
        if sp.issparse(rna_tf_vals):
            rna_tf_vals = rna_tf_vals.todense()
        rna_tf_vals = np.array(rna_tf_vals)

        median_cell = np.median(rna_tf_vals.sum(axis=1))
        # rna_tf_vals = median_cell * (rna_tf_vals / rna_tf_vals.sum(axis=1, keepdims=True))

        # normalise without creating NaNs; zero-sum rows remain zero
        rna_tf_vals = np.divide(
            rna_tf_vals, row_sums, out=np.zeros_like(rna_tf_vals, dtype=float), where=row_sums != 0
        ) * median_cell
        topic_tf = np.array([rna_tf_vals[top_k_indices[:, t], :].mean(axis=0) for t in range(model.num_topics)])
        topic_tf = torch.from_numpy(topic_tf)

        preds_tf_denoised_min, _ = torch.min(topic_tf, dim=1, keepdim=True)
        preds_tf_denoised_max, _ = torch.max(topic_tf, dim=1, keepdim=True)
        topic_tf = (topic_tf - preds_tf_denoised_min) / (preds_tf_denoised_max - preds_tf_denoised_min + 1e-9)
        topic_tf[topic_tf < config_file.tf_expression_clamp] = 0
        topic_tf = topic_tf.to(device)
        return topic_tf

So now, if we the row is not all 0, i.e., we have at least one TF expressed in a topic, we do the calculation, otherwise we put 0. Which makes sense for the rows of 0s.
And then I got this error after Phase 1:

INFO:scdori._core.train_grn:Encoder is now frozen in GRN phase.
INFO:scdori._core.train_grn:Peak-gene links are now frozen in GRN phase.
INFO:scdori._core.train_grn:Topic-peak decoder is now frozen in GRN phase.
INFO:scdori._core.train_grn:Topic-tf decoder is now frozen in GRN phase.
Extracting latent topics: 100%|██████████| 678/678 [06:08<00:00,  1.84it/s]
INFO:scdori._core.train_grn:Starting GRN training
GRN Epoch 0:   0%|          | 0/678 [00:10<?, ?it/s]
Traceback (most recent call last):
  File "/projects/p_scads_spinal_cord/scDORI/30092025/scripts/train.py", line 206, in <module>
    model = train_model_grn(
            ^^^^^^^^^^^^^^^^
  File "/data/horse/ws/dmse952c-scDoRI_env/scDoRI/src/scdori/_core/train_grn.py", line 475, in train_model_grn
    out = model(
          ^^^^^^
  File "/home/dmse952c/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dmse952c/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/horse/ws/dmse952c-scDoRI_env/scDoRI/src/scdori/_core/models.py", line 339, in forward
    C[topic] = tf_expression_input[topic] @ G_act + tf_expression_input[topic] @ G_rep
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: double != float

I think this error is very similar to what you have!
For me it happened because in the np.divide, I put dtype=float for the np.zeros_like array. And apparently it makes dtype of float64 there, while pytorch expects float32.
So, then I added this line: topic_tf = torch.from_numpy(topic_tf).to(dtype=param_dtype), when I convert the topic_tf from numpy to pytorch tensor. This ensures that topic_tf tensor will have the same data type as other stuff in the model:

    if tf_expression_mode == "True":
        latent_all_torch = get_latent_topics(
            model, device, train_loader, rna_anndata, atac_anndata, num_cells, tf_indices, encoding_batch_onehot
        )
        top_k_indices = np.argsort(latent_all_torch, axis=0)[-config_file.cells_per_topic :]
        rna_tf_vals = rna_anndata.X[:, tf_indices]
        if sp.issparse(rna_tf_vals):
            rna_tf_vals = rna_tf_vals.todense()
        rna_tf_vals = np.array(rna_tf_vals)

        median_cell = np.median(rna_tf_vals.sum(axis=1))
        # rna_tf_vals = median_cell * (rna_tf_vals / rna_tf_vals.sum(axis=1, keepdims=True))

        # normalise without creating NaNs; zero-sum rows remain zero; additionally we avoid the division by 0
        row_sums = rna_tf_vals.sum(axis=1, keepdims=True)
        rna_tf_vals = np.divide(
            rna_tf_vals, row_sums, out=np.zeros_like(rna_tf_vals, dtype=float), where=row_sums != 0
        ) * median_cell
        topic_tf = np.array([rna_tf_vals[top_k_indices[:, t], :].mean(axis=0) for t in range(model.num_topics)])
        param_dtype = next(model.parameters()).dtype   # add once near the top of the function if you prefer
        topic_tf = torch.from_numpy(topic_tf).to(dtype=param_dtype)
        # topic_tf = torch.from_numpy(topic_tf)

It helped me and now my losses are all not nan. I have not checked yet if the model trained well and I do not know if my modifications have an effect on the final results. I hope it was helpful! If you have any other questions, feel free to ask!

Best,
Dmitrii

[sbgonenc]

Hi Dmitrii!
I could get back to experimentations with scDoRI, and I should say the code you shared works like a charm! So far I did not get any double != float errors! @saraswatmanu and @martinrohbeck it's easy to implement!
Thanks so much