Feat (benchmark): Allow easy specification of different search algorithms within benchmark scripts

[nickfraser]

Builds on #1356. Introduces a BenchmarkSearchMixin which determines how we generate experiments with the benchmark utils. The first one GridSearchMixin, replicates the behaviour of the benchmark utils before this PR was made.

The second one is a RandomSearchMixin, which allow specification of various search algorithms that can be used to specify parameters, for example:

act_equalization:
  rand_type: choices
  rand_values: [null, "layerwise", "fx"]
act_equalization_alpha:
  rand_type: linear
  rand_values: [0.05, 0.95]
gptq:
  rand_type: const
  rand_values: true
learned_round_lr:
  rand_type: log2
  rand_values: [0.0001, 0.1]
learned_round_scale_lr:
  rand_type: exp2
  rand_values: [0.0001, 0.1]

Running as follows:

python llm_rand_benchmark.py --config benchmark_rand_template.yaml --dry-run --seed 1 --num-experiments 10

Will give the following output:

Num. experiments: 10
Benchmark args.:
        config: benchmark_rand_template.yaml
        results_folder: ./
        gpus: 0
        num_gpus_per_process: 1
        max_num_retries: 1
        dry_run: True
        num_experiments: 10
        max_experimental_configs: 100000
        seed: 1
Non-default args.:
        --act-equalization: type: choices, values: [None, 'layerwise', 'fx']
        --act-equalization-alpha: type: linear, min: 0.05, max: 0.95
        --gptq: type: const, value: True
        --learned-round-lr: type: log2, min: 0.0001, max: 0.1
        --learned-round-scale-lr: type: exp2, min: 0.0001, max: 0.1

Note, a limitation with the current approach is that the worker queue generation (i.e., the random search space) and execution (i.e., the experiments) are two separate steps, meaning that it's difficult to integrate experiment feedback into the search space selection (e.g., for Bayesian optimization or simulated annealing).

[nickfraser]

Switch to Mixin.

[nickfraser]

Let me know if you prefer me to revert this. In my view, this should be fail instead of a warning. Could also be a separate PR.

[pablomlago]

I'm fine being such a small change. Nit:

Suggested change

	assert False, "GPTQ, GPFQ, and/or Qronos are enabled together."
	assert (int(args.gptq) + int(args.gpfq) + int(args.qronos)) <= 1, "GPTQ, GPFQ, and/or Qronos cannot be enabled together."

[pablomlago]

Suggested change

	def standardize_args(cls, script_args: str) -> Dict[str, List]:
	def standardize_args(cls, script_args: Namespace) -> Dict[str, List]:

[pablomlago]

Indicate that the concrete classes need to provide the abstract class variable argument_parser, e.g.

    @property
    @abstractmethod
    def argument_parser(self) -> ArgumentParser:
        pass

[pablomlago]

Suggested change

	def standardize_args(cls, script_args: str) -> Dict[str, List]:
	def standardize_args(cls, script_args: Namespace) -> Dict[str, List]:

[pablomlago]

There is a subset of the functionality of standardize_args which is shared for GridSearchMixin and RandomSearchMixing, e.g. YAML reading. Is it possible to extract the common functionality to the abstract parent class.

[pablomlago]

Most of the arguments are shared between GridSearchMixin and RandomSearchMixin, I would consider extracting the common logic.

[pablomlago]

Can we remove these? I would rather go for having multiple inheritance in the entrypoints, e.g.

class ImagenetPTQBenchmarkUtils(BenchmarkUtils, GridSearchMixin):

[pablomlago]

Maybe it is worth renaming q to something more self-explanatory.