Warm runners

[kichik]

All of our runners are currently provisioned on-demand. That means it can take a minimum of a few seconds until a runner is ready to take on a job. With Windows runners, that can be even longer. You can already use ECS provider with minInstances=1 to greatly reduce that time to just the starting of a container, but ECS is not a good fit for everyone.

• We should have a way to define a set of warm runners that are either always on, or some kind of schedule. They will still be ephemeral. We don't want to deal with or encourage dirty runners.
• We want this to support every provider type, so just an auto-scaling group will not work for providers like Lambda or CodeBuild.
• The step function already accepts idleTimeout that can be used here. We can trigger the step function with a very long idle timeout to spin up a warm runner.
• We probably don't want to stop spinning on-demand runners when there is a warm runner available. They will technically serve as new warm runners. Maybe spin those already with high idleTimeout?
• It would be useful to provide complex scheduling like keeping 10 warm runners during work hours but zero on weekends.
• The interface to define warm runners will have to be aware of org vs repo registration. For repo registration, the user will have to decide which repos the warm runners get assigned to.