Vaultwarden supports ssh keys: https://github.com/dani-garcia/vaultwarden/blob/main/.env.template#L378
I can then use rbw as an SSH Agent.
This would also make installing this flake easier (see #132), since I don´t have to worry about creating to new keys or moving old keys to new machines.