Is it possible to share Rcpp object/pointer between R and Python?

[gregorgorjanc]

I am developing R access to the tskit C API (a library for working with massive genomic datasets by mapping how we relate to each other through our ancestors in the past - https://tskit.dev/). tskit has a very good C API that is further enhanced via a Python API. Both are very complete, well tested, and documented, but the Python API is more popular. I am well aware of reticulate to call Python API from R, but that doesn't suit all our needs(calling the tskit C API within loops in R/Rcpp, hence my effort to bring tskit C API to R/Rcpp).

I have developed an R package tskitr at https://github.com/HighlanderLab/tskitr which includes the tskit's C code and have 1) wrote some Rcpp functions that call the C API and 2) developed library build with instructions for other packages to use tskit C API via Rcpp in their work (the reason why I have built the package in the first place).

While this all works very well thanks to Rcpp, I don't want to duplicate the work done on the Python API. Most users will still be encouraged to analyse the objects using Python API, possibly via reticulate.

While we can build tskit objects on the R side (using Rcpp), export them to disk, load into Python and analyse in standalone or reticulate Python, I wonder if we could share Rcpp object/pointer between R and reticulate Python sessions? Doing a bit of LLMing, I surmise that the answer is no due to ABI incompatibility, but I thought I would ask real human experts! In the meanwhile I have written ts_r_to_py() and ts_py_to_r() that use disk-write-read to transfer between R and reticulate Python. tskit objects are complex so not sure I can figure out r_to_py()/py_to_r() wrappers and I only have a pointer in R session anyway - looking at the C++ code of the reticulate package I get lost very quickly.

Thanks!

[eddelbuettel]

From the top of my head, we should be able to send a pointer a struct around. That is what Arrow does after. But I do not have a working example nor can I think of one -- so what you do by serializing to disk seems like a fair approach. Alternatives would be to write to (in memory?) data bases like duckdb, or (if you remain on Linux) rely on shared memory to access a common object.

All these are great questions and would be really good to have a studied reference example.

[kevinushey]

I'm spitballing, but does tskit provide any APIs for creating a Python object from an existing C pointer to a tskit object? If so, I think you could safely share those pointer addresses between R / Rcpp and Python.

A somewhat bare version of what I'm imagining is the "advanced" Armadillo constructors, whereby you can create an Armadillo matrix from a pointer to auxiliary memory / data. https://arma.sourceforge.net/docs.html#adv_constructors_mat

[eddelbuettel]

Yup. It is a somewhat simpler example as Armadillo 'merely' reuses length * sizeof(x) for either an int or double vector. But the idea of the abstraction is the same: if we share the same C/C++ headers between Python and R so that sizeof(struct foo) is the same than we can instantiate a new object off a pointer. The other thought I had was that instead of serializing to disk as you do you could also look into one of the usual serialization-on-wire examples such as Protocol Buffers, MsgPack, ...

[gregorgorjanc]

I'm spitballing, but does tskit provide any APIs for creating a Python object from an existing C pointer to a tskit object? If so, I think you could safely share those pointer addresses between R / Rcpp and Python.

It's my understanding that Python API is wrapped around the C API, though I don't understand how this is done (I am not a Python developer). Will talk to tskit devs and come back.