Skip to content

feat: eval results #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ki-Seki wants to merge 16 commits into
base: main
Choose a base branch
from
Open

feat: eval results #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
9c54e0d
feat: add some old eval results
Ki-Seki Jan 7, 2026
f5b2e57
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 7, 2026
a5a769a
Merge branch 'main' into feat/eval-results
Ki-Seki Jan 8, 2026
0f879cb
feat: add Match @ Qwen3-1.7B
Ki-Seki Jan 8, 2026
bc7f050
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 8, 2026
7d8f4cc
Merge branch 'main' into feat/eval-results
Ki-Seki Jan 11, 2026
ec310b1
feat: add evaluation script
Ki-Seki Jan 11, 2026
11cd71e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 11, 2026
7c8f021
chore: backup
Ki-Seki Jan 12, 2026
b27f382
chore: backup
Ki-Seki Jan 14, 2026
76dedf3
feat: add evaluation and README scripts for KDD experiments
Ki-Seki Jan 21, 2026
c05a661
Merge branch 'main' into feat/eval-results
Ki-Seki Jan 21, 2026
29c4c1f
fix: update model names in evaluation scripts for consistency
Ki-Seki Jan 21, 2026
c9eef17
chore: backup
Ki-Seki Jan 21, 2026
2950741
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 21, 2026
4167eb9
Merge branch 'main' into feat/eval-results
Ki-Seki Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions results/251009/*README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Initial temporary evals
17 changes: 17 additions & 0 deletions results/251009/aggregated_results.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
filename,total,evaluates,corrects,errors,accuracy,calibrated_accuracy,avg_response_len,start_time,end_time,elapsed_minutes,args,git_commit_id
unsloth_Qwen3-4B-Instruct-2507_Idavidrein_gpqa_251009-221154.json,198,198,46,0,0.23232323232323232,0.23232323232323232,12989.277777777777,2025-10-09T22:11:54.687405,2025-10-09T22:31:48.567086,19.89799468333333,"{'model_name': 'unsloth/Qwen3-4B-Instruct-2507', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': False, 'reason_budget': 0, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",9c94f1618b97f48d104402039d9bb5309e68c05c
unsloth_Qwen3-4B-Instruct-2507_TIGER-Lab_MMLU-Pro_251009-223159.json,200,200,80,0,0.4,0.4,4960.89,2025-10-09T22:31:59.907212,2025-10-09T22:40:00.040106,8.0022149,"{'model_name': 'unsloth/Qwen3-4B-Instruct-2507', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': False, 'reason_budget': 0, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",92951712f4e680a89a25ee5aec7ebbb386996fef
artifacts_09251-gim-sft-tmp_sft-gim_Idavidrein_gpqa_251009-230904.json,198,198,66,5,0.3333333333333333,0.34196891191709844,1880.7575757575758,2025-10-09T23:09:04.234325,2025-10-09T23:12:20.117634,3.2647218166666665,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 2, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_Idavidrein_gpqa_251009-231231.json,198,198,69,6,0.3484848484848485,0.359375,2217.747474747475,2025-10-09T23:12:31.332615,2025-10-09T23:16:23.802934,3.8745053166666663,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 4, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_Idavidrein_gpqa_251009-231633.json,198,198,70,4,0.35353535353535354,0.36082474226804123,2644.2272727272725,2025-10-09T23:16:33.800228,2025-10-09T23:20:41.265066,4.124413966666666,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 6, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_Idavidrein_gpqa_251009-232055.json,198,198,73,9,0.3686868686868687,0.3862433862433862,2952.7979797979797,2025-10-09T23:20:55.911530,2025-10-09T23:26:29.007648,5.5516019666666665,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 8, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_Idavidrein_gpqa_251009-232641.json,198,198,72,7,0.36363636363636365,0.3769633507853403,3255.6363636363635,2025-10-09T23:26:41.204684,2025-10-09T23:32:01.579146,5.339574366666667,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 10, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_Idavidrein_gpqa_251009-233212.json,198,198,69,8,0.3484848484848485,0.3631578947368421,3577.257575757576,2025-10-09T23:32:12.035222,2025-10-09T23:37:51.723390,5.661469466666667,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 12, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_Idavidrein_gpqa_251009-233801.json,198,198,62,7,0.31313131313131315,0.32460732984293195,4011.681818181818,2025-10-09T23:38:01.511545,2025-10-09T23:44:19.553620,6.30070125,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 14, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 198, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'Idavidrein/gpqa', 'name': 'gpqa_diamond', 'split': 'train'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_TIGER-Lab_MMLU-Pro_251009-234433.json,200,200,111,2,0.555,0.5606060606060606,1462.38,2025-10-09T23:44:33.279775,2025-10-09T23:47:06.751190,2.5578569166666667,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 2, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_TIGER-Lab_MMLU-Pro_251009-234717.json,200,200,110,3,0.55,0.5583756345177665,1685.565,2025-10-09T23:47:17.236698,2025-10-09T23:50:09.742725,2.8751004499999997,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 4, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_TIGER-Lab_MMLU-Pro_251009-235020.json,200,200,109,2,0.545,0.5505050505050505,2013.88,2025-10-09T23:50:20.485066,2025-10-09T23:53:04.007364,2.7253716333333333,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 6, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_TIGER-Lab_MMLU-Pro_251009-235315.json,200,200,121,2,0.605,0.6111111111111112,2311.425,2025-10-09T23:53:15.992872,2025-10-09T23:56:37.696530,3.361727633333333,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 8, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_TIGER-Lab_MMLU-Pro_251009-235650.json,200,200,125,1,0.625,0.628140703517588,2575.74,2025-10-09T23:56:50.702339,2025-10-09T23:59:44.885723,2.9030563999999996,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 10, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_TIGER-Lab_MMLU-Pro_251009-235959.json,200,200,116,1,0.58,0.5829145728643216,2891.185,2025-10-09T23:59:59.099936,2025-10-10T00:03:17.430844,3.305515133333333,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 12, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
artifacts_09251-gim-sft-tmp_sft-gim_TIGER-Lab_MMLU-Pro_251010-000327.json,200,200,121,1,0.605,0.6080402010050251,2985.24,2025-10-10T00:03:27.768892,2025-10-10T00:07:01.421817,3.5608820833333334,"{'model_name': 'artifacts/09251-gim-sft-tmp/sft-gim', 'api_key': '', 'base_url': 'http://localhost:8000/v1', 'is_gim': True, 'reason_budget': 14, 'temperature': 0.0, 'presence_penalty': 1.0, 'max_tokens': 8192, 'seed': 16, 'first_n': 200, 'num_proc': 20, 'output_dir': 'results', 'dataset': {'path': 'TIGER-Lab/MMLU-Pro', 'name': None, 'split': 'test'}}",72a79d260d5bcaa1276b46a9192a4e811fe52a3d
Loading