[AutoAgent] Add Validation to AutoKernelAgent routed Models #79
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AutoKernelAgentdoes not validate whether the generated or cachedrouter_cache.jsonentry is requesting models that are unavailable to the caller.This PR introduces
_validate_cfg_modelswhich checks model availability against the provider when writing or loading fromrouter_cache.jsonAlong the way, we also fix a routing bug where
route_cfgis not properly set when there is a cache miss (i.e when router_model generates a new entry).Fuser.auto_agentthe routing config is generated, but not utilized (instead CLI args are used). When the model is called a second time (with a cache hit), the config is then properly utilized resulting in different outputs (first one is wrong, second is right)Test
Added
pytest tests/fuser/test_auto_agent_cache.pyChecking with empty cache
python -m Fuser.auto_agent --problem /home/jackkhuu/KernelBench/KernelBench/level3/33_VanillaRNN.pyManual testing with
router_cache.jsonentry and observing that the cli model is used instead