Dataset experiments#220

Open

motiwari wants to merge 83 commits intomainfrom

dataset_experiments

Commits on Jul 28, 2022

Do experiments on large dataset (not clean)
vxbrandon
committed

Commits on Jul 29, 2022

Handling a few extreme cases!
vxbrandon
committed
Run experiments on various real dataset.
vxbrandon
committed
Large refactor. Moving sklearn and mnist datasets to data_loader in compare_runtimes.py
motiwari
committed
Refactoring datasets in compare_budgets.py
motiwari
committed
Large cleanup: compare_runtimes.py and compare_budget.py now avoid duplicated code. Also cleaned up Jeys real_experiments.py and regression_experiment.py
motiwari
committed
Passing dataset name to store profiles with dataset names and not overwrite
motiwari
committed
Cleaning (deleting) logfiles
motiwari
committed
Changing num_seeds back to 5 for runtime and budget
motiwari
committed
Adding HOUSING dataset, where our approach doesnt work, and renaming some SKLEARN to SKLEARN_REGRESSION to avoid naming collision
motiwari
committed

Commits on Jul 30, 2022

Commits on Jul 31, 2022

Commits on Aug 1, 2022

Made necessary cosmetic changes to produce_tables.py. Use all of its output except use old AIR Table 2 results
motiwari
committed
Nit
motiwari
committed
TAG: EXP_OPEN. Table 3 experiments, budget for classification, look erroneous. Need to rerun
motiwari
committed
Adding small dataset exps for R3
motiwari
committed
Something is still wrong with Table 3. About to debug
motiwari
committed
TAG: EXP_OPEN. Fixing critical bug in classification budget to NOT just run on MNIST
motiwari
committed

Commits on Aug 2, 2022

TAG: EXP_OPEN. Reducing budget by factor of 5 in FLIGHT
motiwari
committed
TAG: EXP_OPEN. Reducing budget by factor of 5 in COVTYPE
motiwari
committed
Rerunning exps with different max_depth with one seed to see perf diffs
motiwari
committed
TAG: EXP_OPEN. Running final table 3
motiwari
committed
Adding proper Table 3 results
motiwari
committed
After an hour, get .8088 with sklearn RFC and .5964 with Hoeffding Tree
motiwari
committed
Don't plot negative value for num queries
vxbrandon
committed
Merge remote-tracking branch 'origin/dataset_experiments' into dataset_experiments
vxbrandon
committed
Recreating final Table 3
motiwari
committed

Commits on Sep 10, 2022

Commits on Sep 13, 2022

Merge branch 'reproducec_experiments' of github.com:ThrunGroup/FastForest into reproducec_experiments

Luke Donghyun Lee
authored and
Luke Donghyun Lee
committed

Commits on Sep 14, 2022

ran all experiments except gpu

Luke Donghyun Lee
authored and
Luke Donghyun Lee
committed
ran all experiments except gpu

Luke Donghyun Lee
authored and
Luke Donghyun Lee
committed

Commits on Sep 15, 2022

Change hyperparams to reproduce experiments on Sklearn regression and MNIST
vxbrandon
committed

Commits on Sep 19, 2022

1. Change back our algorithm to find the best arm not the 5th best arm
vxbrandon
committed

Commits on Oct 9, 2022

Changing mab solver to len(candidates) > 1
motiwari
committed
Fixing repro script
motiwari
committed

Commits on Oct 10, 2022

Adding results, choosing final datasets
motiwari
committed
Fixing table-making code
motiwari
committed
Adding tables
motiwari
committed

Commits on Nov 8, 2022

Commits on Nov 9, 2022

Need to sample randomly 1000 times for roughly equal performances, theirs still slightly worse
motiwari
committed

Commits on Dec 12, 2022

Commits on Dec 13, 2022

All results recreated. Table 1 multipliers are a little more modest with more generous hyperparameters, Table 1 forest covtype accuracies a bit worse, and Table 2 some rows are identical, Scaling p…
motiwari
committed