Optimization results aren't exactly the same from run to run even with set seed

[kelvinburke]

So doing some testing for the optimization-start-skip branch - I found that from run to run, the function values that an optimization would get would not always be exactly the same if you re-run with the same seed.

This is because of:
constrain_sum_bounded() calls scipy.optimize.minimize, which can give a slightly different solution if you re-order the initial vector.

constrain_sum_bounded() is called by TotalSpendConstraint.constrain_instructions() and the hard_constraints that it receives has hard_constraints["programs"][t] as a set. This means from run to run hard_constraints["programs"][t] can change order, so the input to constrain_sum_bounded() changes order and gives a different constrained budget.

So the budget getting constrained differently gives different results from run to run.

Solutions are:

• progs = sorted(hard_constraints["programs"][t]) # Programs eligible for constraining at this time gives a consistent order to the programs from run to run. This alone fixes the problem
• Optionally order the array in constrain_sum_bounded() before running scipy.optimize.minimize and then put back in the original order. This doesn't always fix it - if the values in the array have duplicates. So this change is not necessary and I might undo it?
• Possibly could change hard_constraints["programs"][t] to not be a set? I haven't tried this

See optimization-start-skip branch for these committed changes

[RomeshA]

Thanks @kelvinburke, of those options I think sorting the programs with sorted(hard_constraints["programs"][t]) would be the best solution 👍