rootless container Quadlet with drop-in doesn't work

[opelx]

Issue Description

I followed Template files. Even with the minimal example, I wasn't able to get the expected result: using a drop-in to override the default settings of a Quadlet.

Steps to reproduce the issue

1. Create a container file

$ cat ~/.config/containers/systemd/sleep@.container
[Unit]
Description=A templated sleepy container

[Container]
Image=docker.io/library/alpine:3
Exec=sleep %i

[Service]
Type=oneshot

[Install]
WantedBy=default.target
DefaultInstance=60

2. start it for basic test, which works as expected - sleep 3s

$ systemctl --user daemon-reload && systemctl --user start sleep@3

3. This is expected, even I' not sure about the reason for this

$ systemctl --user enable sleep@3
Failed to enable unit: Unit /run/user/1000/systemd/generator/sleep@.service is transient or generated

4. Create the symbolic link

$ cd ~/.config/systemd/user/
$ ln -s /home/core/.config/containers/systemd/sleep@.container sleep@3.container

5. Create the drop-in dir

mkdir ~/.config/systemd/user/sleep@3.container.d

6. Add the override

$ cat ~/.config/systemd/user/sleep\@3.container.d/override.conf
[Container]
Exec=echo "Won't sleep!"

7. reload and start again, which also sleeps for 3s (which shouldn't)

$ systemctl --user daemon-reload && systemctl --user start sleep@3.service

8. investigating, look into the logs

$  journalctl --user -xeu sleep@3.service
[...]
Dec 08 18:25:18 coreos-test podman[2936493]: 2025-12-08 18:25:18.526665489 +0100 CET m=+0.161738084 container create 8bc9d378644b8bc0b1fba6cfe11dc960457a855e03df1faa702b2a7a4b74ae15 (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 08 18:25:18 coreos-test podman[2936493]: 2025-12-08 18:25:18.460045111 +0100 CET m=+0.095117768 image pull 706db57fb2063f39f69632c5b5c9c439633fda35110e65587c5d85553fd1cc38 docker.io/library/alpine:3
Dec 08 18:25:18 coreos-test podman[2936493]: 2025-12-08 18:25:18.729946012 +0100 CET m=+0.365018770 container init 8bc9d378644b8bc0b1fba6cfe11dc960457a855e03df1faa702b2a7a4b74ae15 (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 08 18:25:18 coreos-test podman[2936493]: 2025-12-08 18:25:18.741037431 +0100 CET m=+0.376110026 container start 8bc9d378644b8bc0b1fba6cfe11dc960457a855e03df1faa702b2a7a4b74ae15 (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 08 18:25:18 coreos-test podman[2936493]: 2025-12-08 18:25:18.755051902 +0100 CET m=+0.390124534 container attach 8bc9d378644b8bc0b1fba6cfe11dc960457a855e03df1faa702b2a7a4b74ae15 (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 08 18:25:21 coreos-test podman[2936493]: 2025-12-08 18:25:21.747101558 +0100 CET m=+3.382174240 container died 8bc9d378644b8bc0b1fba6cfe11dc960457a855e03df1faa702b2a7a4b74ae15 (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 08 18:25:21 coreos-test podman[2936493]: 2025-12-08 18:25:21.922152784 +0100 CET m=+3.557225391 container remove 8bc9d378644b8bc0b1fba6cfe11dc960457a855e03df1faa702b2a7a4b74ae15 (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 08 18:25:22 coreos-test systemd[1149]: Finished sleep@3.service - A templated sleepy container.
[...]

It shouldn't sleep ...

9. Investigate even more:

$ systemd-delta --user sleep@3
systemd-delta: unrecognized option '--user'
$ systemd-analyze --user cat-config sleep@3
Option --user is not supported for cat-config right now.

... no tools I'm aware of work here.

With

$ systemctl --user edit unit --drop-in=sleep@3
No files found for unit.service.
Run 'systemctl edit --user --force --full unit.service' to create a new unit.

I don't know how to use it correctly :-(

Describe the results you received

The Exec in the Quadlet isn't overridden by the drop-in file, or I misunderstood the manual (maybe misread?)

Describe the results you expected

The Exec in the Quadlet isn't overridden by the drop-in file with Exec=echo "Won't sleep!" which shall be seen in the logs.

Podman info output

$ podman version
Client:        Podman Engine
Version:       5.6.2
API Version:   5.6.2
Go Version:    go1.25.1 X:nodwarf5
Git Commit:    9dd5e1ed33830612bc200d7a13db00af6ab865a4
Built:         Tue Sep 30 02:00:00 2025
Build Origin:  Fedora Project
OS/Arch:       linux/amd64
$ podman info
host:
  arch: amd64
  buildahVersion: 1.41.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-2.fc43.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 85.33
    systemPercent: 4.16
    userPercent: 10.51
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "43"
  eventLogger: journald
  freeLocks: 2012
  hostname: coreos-test
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.17.7-300.fc43.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 3368030208
  memTotal: 8313237504
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc43.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.fc43.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.24-1.fc43.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.24
      commit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250919.g623dbf6-1.fc43.x86_64
    version: |
      pasta 0^20250919.g623dbf6-1.fc43.x86_64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-3.fc43.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.6.0
  swapFree: 0
  swapTotal: 0
  uptime: 96h 2m 50.00s (Approximately 4.00 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 16
    paused: 0
    running: 14
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 25163726848
  graphRootUsed: 19890372608
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 23
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.2
  BuildOrigin: Fedora Project
  Built: 1759190400
  BuiltTime: Tue Sep 30 02:00:00 2025
  GitCommit: 9dd5e1ed33830612bc200d7a13db00af6ab865a4
  GoVersion: go1.25.1 X:nodwarf5
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.2
$ rpm -q podman
podman-5.6.2-1.fc43.x86_64

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="43.20251110.3.1 (CoreOS)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=43
VERSION_CODENAME=""
PRETTY_NAME="Fedora CoreOS 43.20251110.3.1"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:43"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=43
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=43
SUPPORT_END=2026-12-02
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='43.20251110.3.1'
IMAGE_VERSION='43.20251110.3.1'

Additional information

This happens to my real use-case, as of the example given.

[mheon]

For your point 3 - that's a systemd issue, nothing we can really do about it. Recommended solution is to use dependencies (like WantedBy) in the service to make it auto-start.

@ygalblum PTAL for the rest.

[opelx]

Thank you very much. I read somewhere that the services generated by Quadlets are considered generated by systemd and, like transient services, cannot be enabled. I don't understand the background to this, but it probably has something to do with how it works internally behind the curtains. For me as a user, container/Podman and systemd somehow come together here, hence I mentioned this.

[opelx]

Maybe the manual mentioned above is not correct?

$ systemctl --user status sleep@3.service
○ sleep@3.service - A templated sleepy container
     Loaded: loaded (/var/home/core/.config/containers/systemd/sleep@.container; generated)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
             /var/home/core/.config/systemd/user/sleep@3.service.d
             └─override.conf
     Active: inactive (dead)

Dec 13 18:51:24 coreos-test podman[1483399]: 2025-12-13 18:51:24.085190691 +0100 CET m=+0.155942825 container create e8d232183cec3f9abd04db35d68ee00db853dca934a038fe6d6507975502ff1d (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 13 18:51:24 coreos-test podman[1483399]: 2025-12-13 18:51:24.018708371 +0100 CET m=+0.089460705 image pull 706db57fb2063f39f69632c5b5c9c439633fda35110e65587c5d85553fd1cc38 docker.io/library/alpine:3
Dec 13 18:51:24 coreos-test podman[1483399]: 2025-12-13 18:51:24.289149513 +0100 CET m=+0.359901698 container init e8d232183cec3f9abd04db35d68ee00db853dca934a038fe6d6507975502ff1d (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 13 18:51:24 coreos-test podman[1483399]: 2025-12-13 18:51:24.299951927 +0100 CET m=+0.370704087 container start e8d232183cec3f9abd04db35d68ee00db853dca934a038fe6d6507975502ff1d (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 13 18:51:24 coreos-test podman[1483399]: 2025-12-13 18:51:24.311532333 +0100 CET m=+0.382284518 container attach e8d232183cec3f9abd04db35d68ee00db853dca934a038fe6d6507975502ff1d (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 13 18:51:27 coreos-test podman[1483399]: 2025-12-13 18:51:27.306561816 +0100 CET m=+3.377313988 container died e8d232183cec3f9abd04db35d68ee00db853dca934a038fe6d6507975502ff1d (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 13 18:51:27 coreos-test podman[1483399]: 2025-12-13 18:51:27.460996669 +0100 CET m=+3.531748828 container remove e8d232183cec3f9abd04db35d68ee00db853dca934a038fe6d6507975502ff1d (image=docker.io/library/alpine:3, name=systemd-sleep_3, PODMAN_SYSTEMD_UNIT=sleep@3.service)
Dec 13 18:51:27 coreos-test systemd[1149]: Finished sleep@3.service - A templated sleepy container.
Dec 13 18:51:31 coreos-test systemd[1149]: /var/home/core/.config/systemd/user/sleep@3.service.d/override.conf:1: Unknown section 'Container'. Ignoring.
Dec 13 18:51:31 coreos-test systemd[1149]: /var/home/core/.config/systemd/user/sleep@3.service.d/override.conf:1: Unknown section 'Container'. Ignoring.

Note: Unknown section 'Container'. Ignoring - the manual systemd.unit shows:

[Container]
Image=quay.io/centos/centos

I don't see any problem with overwriting the Exec= Key instead.

Note, that I have renamed the drop-in directory to sleep@3.service.d opposite as stated in the manual (foo@instance.container.d).

[ygalblum]

The issue is with the directory in which you've placed the symlink and the drop-in directory.

You need to keep in mind that Quadlet is not part of Systemd, it is a Systemd generator. Meaning, it knows to take an input (the different Quadlet files) and transform them into service unit files that Systemd can read. What you did, placing the drop-ins in Systemd's directories, made Systemd try to read a Quadlet file, which it does not know, resulting in the Unknown section 'Container'. Ignoring message.

Instead, the symlink and drop-in directory should be placed in the Quadlet directory. This will make Quadlet handle the drop-in.

In you example, the directory is ~/.config/containers/systemd:

$ cd ~/.config/containers/systemd
$ ln -s sleep@.container sleep@3.container
$ mkdir sleep@3.container.d
$ cat sleep@3.container.d/override.conf
[Container]
Exec=echo "Won't sleep!"

Please let me know your findings.

BTW you are correct. The Systemd unit files created by Quadlet are considered by Systemd as transient and as a result cannot be enabled. See: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html#enabling-unit-files

[opelx]

Thank you for your answer!

# see above
$ systemctl --user daemon-reload
$ systemctl --user status sleep@3.service
○ sleep@3.service - A templated sleepy container
     Loaded: loaded (/var/home/core/.config/containers/systemd/sleep@3.container; generated)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
...
$ systemctl --user status sleep@3.container
○ sleep@3.container.service - A templated sleepy container
     Loaded: loaded (/var/home/core/.config/containers/systemd/sleep@.container; generated)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
$ systemctl --user start sleep@3.service
$ journalctl --user -xeu sleep@3.service
...
Dec 15 17:15:52 coreos-test systemd-sleep_3[2230695]: Won't sleep!
...

So, it works as expected. The error was that I put the drop-in into ~/.config/systemd/user. The manual should be clear about the place/location of Quadlets' drop-in. Anyway, there doesn't seem to be a clear way to see the success of integration of the drop from systemd site, does it?

$ systemctl --user cat sleep@3.service
...
[X-Container]
Image=docker.io/library/alpine:3
Exec=sleep %i
Exec=echo "Won't sleep!"
...

Probably I have to clear the previous command like

Exec=
Exec=echo "Won't sleep!"

since this is transformed into ExecStart=?

[ygalblum]

The manual should be clear about the place/location of Quadlets' drop-in

The reason the Quadlet doc does not specify the directory is that it supports multiple different directories and specifies all of them seemed redundant. Since these are Quadlet drop-ins, not Systemd drop-ins, their base path should be that of Quadlet, not of Systemd.

Probably I have to clear the previous command like

You are again confusing between Quadlet files and Systemd service files. The Exec field you are (correctly) setting belong to the Container section of the .container file. Most of what Quadlet does is transforming the keys passed in the dedicated section (e.g. Container), into the command placed in ExecStart. So, the change to the Exec key in the Container section is correctly put into the ExecStart key of the generated systemd unit file.
Specifically for Exec, since it is defines as a single instance key, Quadlet knows to take the last occurrence and you don't need to clear it.