wrong mapping of "datadir" variable

[skoudoro]

Hi meson-python team,

I encountered this issue when working on dipy/dipy#2715.

With the help of @eli-schwartz, we saw that get_option('datadir') does not return the share folder. Meson-python maps "datadir" to /usr instead of /usr/share -- or more generally, to the python prefix instead of python prefix + "/share".

Thank you in advance for the future fix and let me know if my issue is not clear. Thanks!

[eli-schwartz]

meson-python/mesonpy/__init__.py

Lines 80 to 92 in 8b0733f

	# Map Meson installation path placeholders to wheel installation paths.
	# See https://docs.python.org/3/library/sysconfig.html#installation-paths
	_INSTALLATION_PATH_MAP = {
	'{bindir}': 'scripts',
	'{py_purelib}': 'purelib',
	'{py_platlib}': 'platlib',
	'{moduledir_shared}': 'platlib',
	'{includedir}': 'headers',
	'{datadir}': 'data',
	# custom location
	'{libdir}': 'mesonpy-libs',
	'{libdir_shared}': 'mesonpy-libs',
	}

Files installed to meson's --datadir directory are inserted into the wheel as such:

• meson install: /usr/share/foo.txt
• wheel file: {distribution}-{version}.data/data/foo.txt

The wheel file then maps this to sysconfig.get_path('data') which is defined as '{base}' (and hence e.g. /usr rather than /usr/share).

[dnicolodi]

As @eli-schwartz points out, meson-python is not responsible for defining where the components of a wheel package are installed. That is part of the wheel package format standard and ultimately it is responsibility of the Python package installer to place the wheel components in the right location. The support for data files in wheel is vestigial (had been carried over from the older egg format, but it is not very well defined). In particular there isn't a good way to know at run time where the data files have been installed.

There is no way for meson-python to instruct pip to install data files in /usr/share/. Even if there would be one, it would not work on Windows where there is no such concept, or for wheel installed in Python virtual environment, where the path would need to be within the virtual environment root.

For this reason, the use of the data directory in wheel packages is discouraged. The portable thing to do is to install the data files alongside the package code in {purelib} or {platlib} and use importlib.resources to access them. We document this here https://meson-python.readthedocs.io/en/latest/reference/limitations.html#non-package-data-files

[eli-schwartz]

No, what I'm pointing out is that the wheel package format has well-defined this to be "plop those files directly into the sys.prefix of your python interpreter, whatever that may be".

There is almost no practical use other than to have files in {distribution}-{version}.data/data/share/ but that is beside the point.

It is often wrong, of course, because typically you must install files to either /usr/share, /usr/local/share, or ~/.local/share and installing them to ~/git/myproject/currentvenv/share/ fails to accomplish any goal whatsoever as the files won't be picked up by system integration. That is why it's discouraged, because "anything that doesn't work in a venv isn't portable" (apparently).

But meson-python is still mishandling what definition is there.

[dnicolodi]

I now see what you mean. However, I'm not convinced that prepending share to the data files location is the correct thing to do. For example, it would not make much sense on Windows, would it? What do other build backed do?

[eli-schwartz]

From meson's side, the distinction is that wheel "data" corresponds to meson's get_option('prefix') rather than datadir. Windows users could certainly configure prefix as "C:/myapp" and datadir as "resources", thus resulting in files installed to "C:/myapp/resources/".

Meson-python could bundle up all files installed anywhere within prefix, or just the ones that are in the datadir subset. I'm not sure what the answer should be.

We do, after all, agree that wheel "data" doesn't really specify what it should mean, regardless of how well defined its install scheme key defines the output directory pathname.

[eli-schwartz]

What do other build backed do?

Setuptools is the only other build system that I happen to know offhand supports it. And setuptools requires you to list files as "data files: collection of paths that may or may not not begin with a forward slash, and if they begin with a forward slash then you cannot use pep517 because it will just arbitrarily do something that makes no sense, and if they begin without a forward slash, and should be installed to /usr/share/foo.txt, then define the path as share/foo.txt".

As part of setuptools' general move to stop worrying about being a build system for python software and start being an interchangeable widget "for installing libraries and cli entrypoints" indistinguishable from any other build backend, they've (soft?) deprecated their support for the wheel "data" scheme because they don't want to deal with people trying to use it and then wondering why Linux GUI software installed in a virtualenv doesn't have any Desktop Menu integration, icon schemes, mimetypes, or man pages and cannot be used at all. Personally, I think the answer is that people should stop using virtualenvs, but what do I know. 🤷

[rgommers]

There is almost no practical use other than to have files in {distribution}-{version}.data/data/share/ but that is beside the point.

There's more uses, e.g. mkl (and mkl-devel, mkl-include) use this to install to <prefix>/lib and <prefix>/include. And pybind11-global installs to both <prefix>/include and <prefix>/share.

Virtualenvs are a headache indeed - on the one hand I'm not a fan, but on the other hand if we start supporting this we may give users something that doesn't work as they expect on Windows nor with virtualenvs. There are still use cases left then (e.g., I'm using mkl from PyPI in a Linux CI job myself), but it's a double-edged sword.

[noamraph]

Hi, @rgommers referred me to here, after I opened issue #835 and PR #836. I really want to make Meld installable in a venv, which would also allow it to be run by uvx meld or similar.

May I suggest a solution which wouldn't break anything? Perhaps add an option to meson-python that would install everything under prefix (which isn't under bindir, py_puredir etc) into the Python installation scheme prefix (whose unique identifier is "data", but really has nothing to do with meson's datadir). If we rely on an option, we won't break anything.

So, we would keep the current behavior, unless you add something like this to pyproject.toml:

[tool.meson-python]
install-everything-in-prefix = true

I would be happy to update my PR to have this. What do you think?