Improve scenario migration safety and process #1666
Description
Description
At the moment, it is hard to test scenario migrations 100%, because we do not have a stable dump of all migratable scenarios on beta/pro. That means that edge cases which were not considered during development can cause the migration to fail halfway through, leading to complications like twice-migrated scenarios.
Current Behavior
We currently have a class, ScenarioMigration, which filters for keep_compatible scenarios, and provides useful output about how many scenarios where migrated etc. This method also raises an error if no scenarios were actually effected by the given migration.
Proposed Improvement
The easiest improvement would be enhanced logging:
- 1. More verbose output in the case of failures - it would be helpful to at least know which scenarios were the last to be migrated, and to know that the order of migration is oldest to newest (older scenarios are more likely to break, and generally less likely to be important).
More complicated, but to give us certainty with our migrations would be to add a 'dry-run' option:
- 2. Basically skip the scenario.save step for each scenario, but add more logging in some sort of handy format (store in a tmp file or log directly to logs).
So to test a migration on beta for example, I would push the migration to beta (without schema changes) and then trigger the dry run on beta with a rake/runner task. If happy with the results, just retrigger the task with dry-run:false and the migration will run for real.
Success Criteria
- It's harder to make a mistake when running a migration
- When migrations do go wrong, it's easy to identify why and resolve the issue