Skip to content

Conversation

@Jack-Khuu
Copy link
Contributor

@Jack-Khuu Jack-Khuu commented Jan 16, 2026

AutoKernelAgent does not validate whether the generated or cached router_cache.json entry is requesting models that are unavailable to the caller.

This PR introduces _validate_cfg_models which checks model availability against the provider when writing or loading from router_cache.json


Along the way, we also fix a routing bug where route_cfg is not properly set when there is a cache miss (i.e when router_model generates a new entry).

  • This presents as a subtle bug where the first time a problem is used in Fuser.auto_agent the routing config is generated, but not utilized (instead CLI args are used). When the model is called a second time (with a cache hit), the config is then properly utilized resulting in different outputs (first one is wrong, second is right)
route_cfg = cache[code_hash].get("config") or {}

Test

  • Added pytest tests/fuser/test_auto_agent_cache.py

  • Checking with empty cache python -m Fuser.auto_agent --problem /home/jackkhuu/KernelBench/KernelBench/level3/33_VanillaRNN.py

  • Manual testing with router_cache.json entry and observing that the cli model is used instead

    {'fuser_dispatch_jobs': 'auto', 'compose_max_iters': 3, 'fuser_verify': True, 'llm_models': {'extract': 'gpt-4o', 'dispatch': 'gpt-4o', 'compose': 'gpt-4o'}
    

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jan 16, 2026
@Jack-Khuu Jack-Khuu requested review from Laurawly and kaiming-cheng and removed request for kaiming-cheng January 16, 2026 23:48
@Jack-Khuu Jack-Khuu changed the title Add Validation to AutoRouted Models [AutoAgent] Add Validation to AutoKernelAgent routed Models Jan 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants