Continue to reopen DB and enum changes until DB reopens up-to-date

[tsutsu]

Currently, indexes make an initial read of the database state and the index state, and then work to bring the index up-to-date with that sequence-point of the database. Once the index is caught up with this sequence-point, the indexing process halts (excepting the case where another consumer has requested the index state while the index updater was running; in that case, the indexing process finishes, then notices that another reader is waiting and starts again, reading off another sequence-point and working toward that.)

With this PR, the index updater will now repeatedly reopen the DB and check to see whether it has truly caught up, getting a new sequence-point to work against each time, all within the context of a single index update operation. This means that a synchronous index update (as is used when querying the index state) will only return when the index has been made fully current, rather than just having caught up to the point the DB was at when the index-update began.

COUCHDB-2955
BugzID: 14036

[kxepal]

Hi! Thanks for contribution. This looks important, but how to test that it works and right?

[janl]

Doesn’t this also change fundamentally how views work? And doesn’t this introduce the same race condition we have for compaction, where heavy database writes can stall out the compaction swap?

[davisp]

This turned out to be an awfully big change. In hindsight I don't think we're going to get much benefit out of reopening the database (I made that assumption almost four years ago so forgive me). Though what might be useful is a check in the handle_info clause when we get the updated index state. Something like this maybe: https://gist.github.com/davisp/153c239cfcd8a4dc53be2e43e84d280d

@janl I'm not sure what you mean by fundamentally changes how views work. This change just makes it so that we restart the indexer rather than wait for the message that a client is still waiting. The technical difference would be that if you had a single view read during db updates, it would update until all active updates were handled instead of handling an arbitrary number of them and waiting for another view read.

Views have always been able to stall the compaction swap. I guess this would lessen the read load to hit that condition though.