Error: Cannot convert a partially known TensorShape to a Tensor: (1, ?)

[kmh005]

Hi,
I used this tool in the past in an older version of R and Python (3.6.2 and 3.6.3 respectively), and it worked like a charm, and thanks for your input to my imbalanced marker set question.
Given the times I'm trying a new install of CellAssign in R 4.1.0 with Python 3.8.3. I originally tried the TF/TF probability install with default versions in a fresh conda environment, but got the below error, so I started a new conda environment and went with an older TF/TF probability 2.1.0. That didn't change the error, below.

I start the session as follows, and check tf:config() to a warning about TensorRT but otherwise successful load, proceed with the workflow below, and include my sessionInfo().

My SCE object is 2449 cells and 157 genes after marker filtering. I used the same 157 markers on a 6k cell object previously to no issues.

Any help or pointers you could provide would be greatly appreciated. Thanks!

library(reticulate)
use_virtualenv("r-reticulate3")
library(tensorflow)
tensorflow::tf_config()
2021-11-22 14:57:53.202921: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; (LD_LIBRARY_PATH...)
2021-11-22 14:57:53.203874: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; (LD_LIBRARY_PATH...)
2021-11-22 14:57:53.203963: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Loaded Tensorflow version 2.1.0
TensorFlow v2.1.0 (~/.conda/envs/r-reticulate3/lib/python3.7/site-packages/tensorflow)


Loaded Tensorflow version 2.1.0
TensorFlow v2.1.0 (~/.conda/envs/r-reticulate3/lib/python3.7/site-packages/tensorflow)
Python v3.7 (~/.conda/envs/r-reticulate3/bin/python3)

fit <- cellassign(exprs_obj = sce[rownames(sce.marker),],
+                      marker_gene_info = sce.marker,
+                      s = sce.s,
+                      learning_rate = 1e-2,
+                      shrinkage = TRUE,
+                      verbose = T,
+                      min_delta=0.25,
+                      num_runs = 10)
Error in py_call_impl(callable, dots$args, dots$keywords) :
  ValueError: Tried to convert 'shape' to a tensor and failed. Error: Cannot convert a partially known TensorShape to a Tensor: (1, ?)

Detailed traceback:
  File "/n/home06/USER/.conda/envs/r-reticulate3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 193, in reshape
    result = gen_array_ops.reshape(tensor, shape, name)
  File "/n/home06/USER/.conda/envs/r-reticulate3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 7443, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/n/home06/USER/.conda/envs/r-reticulate3/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 486, in _apply_op_helper
    (input_name, err))
In addition: Warning message:
In cellassign(exprs_obj = sce[rownames(sce.marker), ], marker_gene_info = sce.marker,  :
  You have specified 157 input genes. Are you sure these are just your markers? Only the marker genes should be used as input

sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /usr/lib64/libblas.so.3.4.2
LAPACK: /usr/lib64/liblapack.so.3.4.2

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel  stats4    stats     graphics  grDevices utils     datasets
[8] methods   base

other attached packages:
 [1] forcats_0.5.1               stringr_1.4.0
 [3] purrr_0.3.4                 readr_1.4.0
 [5] tidyr_1.1.4                 tibble_3.1.5
 [7] ggplot2_3.3.5               tidyverse_1.3.1
 [9] dplyr_1.0.7                 scran_1.20.1
[11] scuttle_1.2.1               cellassign_0.99.21
[13] SingleCellExperiment_1.14.1 SummarizedExperiment_1.22.0
[15] Biobase_2.52.0              GenomicRanges_1.44.0
[17] GenomeInfoDb_1.28.4         IRanges_2.26.0
[19] S4Vectors_0.30.2            BiocGenerics_0.38.0
[21] MatrixGenerics_1.4.3        matrixStats_0.61.0
[23] SeuratObject_4.0.2          Seurat_4.0.4
[25] tensorflow_2.7.0            reticulate_1.22

loaded via a namespace (and not attached):
  [1] readxl_1.3.1              backports_1.2.1
  [3] plyr_1.8.6                igraph_1.2.7
  [5] lazyeval_0.2.2            splines_4.1.0
  [7] BiocParallel_1.26.2       listenv_0.8.0
  [9] scattermore_0.7           tfruns_1.5.0
 [11] digest_0.6.28             htmltools_0.5.2
 [13] fansi_0.5.0               magrittr_2.0.1
 [15] ScaledMatrix_1.0.0        tensor_1.5
 [17] cluster_2.1.2             ROCR_1.0-11
 [19] limma_3.48.3              globals_0.14.0
 [21] modelr_0.1.8              spatstat.sparse_2.0-0
 [23] colorspace_2.0-2          rvest_1.0.2
 [25] rappdirs_0.3.3            ggrepel_0.9.1
 [27] haven_2.4.1               crayon_1.4.1
 [29] RCurl_1.98-1.3            jsonlite_1.7.2
 [31] spatstat.data_2.1-0       survival_3.2-11
 [33] zoo_1.8-9                 glue_1.4.2
 [35] polyclip_1.10-0           gtable_0.3.0
 [37] zlibbioc_1.38.0           XVector_0.32.0
 [39] leiden_0.3.9              DelayedArray_0.18.0
 [41] BiocSingular_1.8.1        future.apply_1.8.1
 [43] abind_1.4-5               scales_1.1.1
 [45] DBI_1.1.1                 edgeR_3.34.1
 [47] miniUI_0.1.1.1            Rcpp_1.0.7
 [49] viridisLite_0.4.0         xtable_1.8-4
 [51] spatstat.core_2.3-0       dqrng_0.3.0
 [53] rsvd_1.0.5                metapod_1.0.0
 [55] htmlwidgets_1.5.4         httr_1.4.2
 [57] RColorBrewer_1.1-2        ellipsis_0.3.2
 [59] ica_1.0-2                 pkgconfig_2.0.3
 [61] dbplyr_2.1.1              uwot_0.1.10
 [63] deldir_1.0-6              locfit_1.5-9.4
 [65] utf8_1.2.2                here_1.0.1
 [67] tidyselect_1.1.1          rlang_0.4.12
 [69] reshape2_1.4.4            later_1.3.0
 [71] cellranger_1.1.0          munsell_0.5.0
 [73] tools_4.1.0               cli_3.1.0
 [75] generics_0.1.1            broom_0.7.10
 [77] ggridges_0.5.3            fastmap_1.1.0
 [79] goftest_1.2-3             fs_1.5.0
 [81] fitdistrplus_1.1-6        RANN_2.6.1
 [83] pbapply_1.5-0             future_1.22.1
 [85] nlme_3.1-152              sparseMatrixStats_1.4.2
 [87] whisker_0.4               mime_0.12
 [89] xml2_1.3.2                rstudioapi_0.13
 [91] compiler_4.1.0            plotly_4.9.4.1
 [93] png_0.1-7                 spatstat.utils_2.2-0
 [95] reprex_2.0.1              statmod_1.4.36
 [97] stringi_1.7.5             lattice_0.20-44
 [99] bluster_1.2.1             Matrix_1.3-4
[101] vctrs_0.3.8               pillar_1.6.3
[103] lifecycle_1.0.1           spatstat.geom_2.3-0
[105] lmtest_0.9-38             RcppAnnoy_0.0.19
[107] BiocNeighbors_1.10.0      data.table_1.14.2
[109] cowplot_1.1.1             bitops_1.0-7
[111] irlba_2.3.3               httpuv_1.6.3
[113] patchwork_1.1.1           R6_2.5.1
[115] promises_1.2.0.1          KernSmooth_2.23-20
[117] gridExtra_2.3             parallelly_1.28.1
[119] codetools_0.2-18          MASS_7.3-54
[121] assertthat_0.2.1          rprojroot_2.0.2
[123] withr_2.4.2               sctransform_0.3.2
[125] GenomeInfoDbData_1.2.6    hms_1.1.0
[127] mgcv_1.8-35               grid_4.1.0
[129] rpart_4.1-15              beachmat_2.8.1
[131] DelayedMatrixStats_1.14.3 Rtsne_0.15
[133] lubridate_1.8.0           shiny_1.7.1
[135] base64enc_0.1-3

[RuiyuRayWang]

Hi @kmh005, I had the exact same issue.

About a year ago I used cellassign in R3.6.x and had great results. Then I put aside the project for a while to work on something else.
Recently I come back to use cellassign and attempted to re-run the pipeline with some new data. Now my R is upgraded to 4.0.3. I received the exact warnings and errors as you do.

I managed to get rid of the TensorRT warning by re-configuring cuda and tensorflow versions.
I had cuda 11.0 installed which requires Tensorflow>=2.4.0. Inspired by this thread, I upgraded tensorflow and tensorflow-gpu to 2.4.0, and the warning was resolved:

# Create independent conda env and install tensorflow(-gpu, -probability) from shell
conda create -n cellassign
conda activate cellassign
pip install --upgrade pip
pip install tensorflow==2.4.0
pip install tensorflow-gpu==2.4.0
pip install tensorflow-probability==0.12.0

Then in R:

> install.packages("tensorflow")
> reticulate::use_condaenv("cellassign")
> library(tensorflow)
> tensorflow::tf_config()
2021-12-02 14:24:33.313201: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Loaded Tensorflow version 2.4.0
TensorFlow v2.4.0 (~/miniconda3/envs/cellassign/lib/python3.7/site-packages/tensorflow)
Python v3.7 (~/miniconda3/envs/cellassign/bin/python)

The cuda & tensorflow compatibility can be found here in case you need it.

But still, running cellassign on example dataset raised error. So TensorRT was not at the center of the issue.

> library(cellassign)
> data(example_sce)
> data(example_marker_mat)
> s <- SingleCellExperiment::sizeFactors(example_sce)
> fit <- cellassign(exprs_obj = example_sce[rownames(example_marker_mat),], 
+                   marker_gene_info = example_marker_mat, 
+                   s = s, 
+                   learning_rate = 1e-2, 
+                   shrinkage = TRUE,
+                   verbose = FALSE)

Error in py_call_impl(callable, dots$args, dots$keywords) : 
ValueError: Tried to convert 'shape' to a tensor and failed. 
Error: Cannot convert a partially known TensorShape to a Tensor: (1, ?)

I also tried reinstalling R 3.6 and the issue persists.

[RuiyuRayWang]

@kmh005 Possible solution: install R tensorflow package via devtools and explicitly specify a version. #94

[WangDaMiao97]

I tried to install a lower version of R and tensorflow. Unfortunately, the issue still occurred.

Following the issue tip, I modified line 165 of the inference-tensorflow.R file, and then, it worked.
p_y_on_c_norm <- tf$reshape(tf$reduce_logsumexp(p_y_on_c_unorm, 0L), as_tensor(shape(1,NULL)))

[sbwiecko]

I tried to install a lower version of R and tensorflow. Unfortunately, the issue still occurred.

Following the issue tip, I modified line 165 of the inference-tensorflow.R file, and then, it worked. p_y_on_c_norm <- tf$reshape(tf$reduce_logsumexp(p_y_on_c_unorm, 0L), as_tensor(shape(1,NULL)))

The current solution is therefore:

1. fork the Irrationene/cellassign repository
2. modify line 165 in inference-tensorflow.R as fixed by @WangDaMiao97, and push it
3. run devtools::install_github("your_fork/cellassign")

That's it, nothing less, nothing more. No need for local Python environment or downgrading to another version of TensorFlow.

sessionInfo()

R version 4.2.0 (2022-04-22 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22621)

other attached packages:
cellassign_0.99.21
tensorflow_2.9.0