panic: error running migrations: failed to delete index key 07935dc9483d36f24464b36e6c000001b9: Txn is too big to fit into one request

[Sakrecoer]

Upgrading to v1.0.6 fails. Haven fails during migrations and won't restart.

Output from journalctl:

Aug 11 11:59:09 basspistol2 haven[780062]: 2025/08/11 11:59:09 [badger] migration 5: delete all indexes and recreate them
Aug 11 11:59:10 basspistol2 haven[780062]: panic: error running migrations: failed to delete index key 07935dc9483d36f24464b36e6c000001b9: Txn is too big to fit into one request
Aug 11 11:59:10 basspistol2 haven[780062]: goroutine 1 [running]:
Aug 11 11:59:10 basspistol2 haven[780062]: main.initRelays()
Aug 11 11:59:10 basspistol2 haven[780062]:         /home/haven/setto/init.go:84 +0x31ba
Aug 11 11:59:10 basspistol2 haven[780062]: main.main()
Aug 11 11:59:10 basspistol2 haven[780062]:         /home/haven/setto/main.go:38 +0x2f0
Aug 11 11:59:10 basspistol2 systemd[1]: haven-setto.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Happy to provide more information if anyone can help me find out where to look

[aaccioly]

Hey @Sakrecoer, this is unfortunately a bug in an upstream Khatru eventstore migration that we can't do much about at the moment. See fiatjaf/eventstore#47 for more details.

This isn't a great workaround, but you can stop Haven, move the DB folder somewhere else, and run /haven --import again to reimport your notes from other relays. Maybe wait for the v1.1.0 release later today which improves the import process🙂.

Apologies for not having better news for now. Features to allow Haven to create better backups (and to reimport them) are planned for future versions.

[Sakrecoer]

Many thanks for your prompt response, @aaccioly ! And no worries, I'm really purely grateful for your responsiveness, and the great software! 💜

FYI, Migration worked fine on another instance running on the same machine. I'll probably wait until v1.1.0 see if i have better luck.

A small question though: will i be losing my blossom files if i run the work around?

[aaccioly]

Thanks for the kind words!
The bug manifests itself on instances with more events. Essentially, when dropping and repopulating an index, the transaction size becomes too large and BadgerDB chokes. IMO, if your system has the resources to run it, LMDB is the better data storage option. Support for SQLite and PostgreSQL is also on the way. While these are not the best fit for relays, they are much easier to evolve and work around if we ever face this sort of problem again.

You won’t lose your Blossom files; those are stored in the filesystem (blossom folder). You may lose some metadata stored in pseudo kind 24242 events (https://khatru.nostr.technology/core/blossom#tracking-blob-metadata), but this isn’t particularly important in Blossom land. Your files will still be served to frens without any issues. The main consequence is that you might have to manually delete old blobs from the blossom folder instead of using Blossom’s API if you ever want some of your older files removed.

I’m also mitigating this issue (and the exposure of those pseudo kind 24242 events over Nostr) by moving Blossom metadata to its own database in v1.1.0.

If you want to try v1.1.0 right now, you can get it here: https://github.com/aaccioly-open-source/haven/releases. Other than a few commits to remove Docker-related stuff and improve documentation, this is exactly what will be in the repo later today.

[dergigi]

Same issue for me:

badger 2025/08/18 14:36:46 INFO: Deleting empty file: db/private/000033.vlog
2025/08/18 14:36:46 [badger] migration 5: delete all indexes and recreate them
panic: error running migrations: failed to save index for event 532395117184ada25bb7e44f99feb1cd393d60862af23ad9b958f9b0a89a4a8b on migration 5: Txn is too big to fit into one request

goroutine 1 [running]:
main.initRelays()
	/home/haven/haven/init.go:72 +0x31d7
main.main()
	/home/haven/haven/main.go:38 +0x2f0

---

I used the mv db db.bak && ./haven --import workaround as described above and that seems to have worked, but for at least one of my haven instances there might be an edge case where certain events are on that relay and that relay only.

Not a big deal, just thought I'd let you know.

[aaccioly]

Thanks for reporting this @dergigi. And yes, I've lost a few notes myself. I apologise for what happened to you and other users.

Unfortunately, this is a limitation of Badger that I don't know how to work around. @fiatjaf had to drop and recreate an existing index on Khatru's Event Store to fix a bug related to querying multiple tags at once. Given how many events some older Haven instances had (especially for folks using Haven since before v1.0.3, when all versions of replaceable items were being stored), dropping the index alone is enough to fill the available transaction space and trigger the error above. Falling back to manual transactions doesn’t help, since deleting such a large index by itself is already too big of an operation. I’ve forked Khatru and EventSyore myself and spent quite a bit of time trying to fix this migration, but unfortunately I haven’t found a clean way around the issue. We would need someone that is properly acquainted with Badger internals on this one.

I’m now working on the Haven v1.1.0 release, and once this is out (hopefully soon, as I just solved the last blocker), I’ll bump the priority of two features that should help Haven become more resilient against this kind of problem:

1. Add support for SQLite and PostgreSQL. Migrations and schema evolution are better understood in relational databases, and I’m personally more familiar with them if something like this happens again. PostgreSQL, for instance, would make these operations much safer. LMDB and Badger are a great fit for relays, but it’s good to give advanced users the choice.
2. Improve Haven backups and implement restore functionality.

I’ll also add a note to the release notes and README so users upgrading from older versions are aware that they may lose data.

Since you’ve already faced the migration issue, it’s likely the upgrade to v1.1.0 will go smoothly. But just in case, I recommend backing up your database before upgrading. Stopping Haven and copying the db folder should be enough, but Badger also has a version-agnostic backup tool that may be useful (though I haven’t tested it myself): https://docs.hypermode.com/badger/quickstart#database-backup

[derekross]

i just tried to upgrade to v1.1.0 and have this issue still.

2025/08/27 13:49:01 🚀 HAVEN v1.1.0 is booting up
badger 2025/08/27 13:49:01 INFO: All 24 tables opened in 1ms
badger 2025/08/27 13:49:01 INFO: Discard stats nextEmptySlot: 0
badger 2025/08/27 13:49:01 INFO: Set nextTxnTs to 271425
badger 2025/08/27 13:49:01 INFO: Deleting empty file: db/private/000063.vlog
2025/08/27 13:49:01 [badger] migration 5: delete all indexes and recreate them
panic: error running migrations: failed to delete index key 01673d5e48000098cd: Txn is too big to fit into one request

goroutine 1 [running]:
main.initRelays()
/home/haven/haven/init.go:75 +0x3205
main.main()
/home/haven/haven/main.go:38 +0x34a

[Sakrecoer]

@derekross As far as I understand, there is no fix for this atm. But the workaround does the cut, given that you have your notes stored somewhere else for haven to fetch them with the --import flag. See #89 (comment)

[aaccioly]

Yeah, this is unfortunately a #wontfix for the reasons already stated. I'll keep this issue open as I've already closed two other issues around this, and despite adding information about the problem to the release notes and README, other folks will open duplicates if I close it.

I think the last "feature" I'll add to Haven v1 will be an improved backup tool and a restore-from-backup tool. Fiatjaf is making some big changes to Khatru (see fiatjaf/khatru#58), and it looks like backup and restore will be the only way to preserve user data when Haven unavoidably migrates to "new" Khatru.

I'm also strongly considering changing Haven v2 to default to SQLite, but that's something to be discussed with @barrydeen for the future. For now, for folks that have the hardware for it, I would strongly recommend migrating to lmdb instead of badger.

[aaccioly]

Just a hint for anyone else struggling with this: you can always run two versions of Haven on different ports side by side, for example, one on version v1.0.5 with your old database and one on version v1.1.0 with a fresh database. Then you can add the "old" instance to relays_import.json and run haven --import so that you can at least preserve a copy of notes that haven't been published to any other relays.

[derekross]

This is an amazing idea. I will do this since the import is not grabbing all of my notes. Thanks