TAAR performance needs to be measured automatically #113
Description
The fullstack call time for TAAR seems to have gone up substantially - or at least it's much more variable comparing TAARv2 and TAARv3
Screenshots below from New Relic show very consistent performance ~70 ms for TAARv2 and variation between 60ms to 120ms for TAARv3.
I suspect this is the probably because of the way we're generating the random sample of add-on suggestions from the whitelist. We currently randomly shuffle the whitelist and pop items off the end which isn't the smartest thing in the world.
Performance profiling will give us definitive answers to this but as a first attempt, we should probably just store an index into the shuffled whitelist array and just treat the array as a circular buffer and read items off as random add-on suggestions.
More importantly, we should try to get an automated performance benchmark where we can spin up webheads in the dev enviroment and throw a load test and snapshot the metrics out of New Relic so we don't have surprise regressions in performance.
