Idea: Use small number of samples from each client #15
Description
We have been recommending using a single sample per interval for each client IP. However, for high test rate clients, this still allows significant bias relative to single test clients, e.g. if we allow one sample per day.
Also, using the median (or geomean) masks the noise, which would be better represented by allowing several semi-random samples from each client.
An alternative might be to take a subsample. For clients that test at least 10 times per month, we might take 3 to 5 samples per month, chosen based on rank, or quantile. If we want 3 samples, we probably should take the medians of the 3 3rds, which would be the 1/6, 3/6, and 5/6, or 17th, 50th, and 83rd percentiles (roughly). In BQ, we could use APPROX_QUANTILE(x, 5)[1,2,3]
We might actually want to choose individual samples, rather than trying to aggregate. To do this, we could order all the samples from a client by BW, and then choose appropriate indices from them, perhaps with a WHERE clause?
If we want one per week, we could randomly choose from among a subset of the samples.