Skip to content

Conversation

@doxav
Copy link
Contributor

@doxav doxav commented May 20, 2025

  1. New PYTEST/CI fully implemented
  • CI uses Phi4-mini with ollama to run additional tests for optimizers with real LLM (moved tests requiring real LLM out of unit_tests to llm_optimizers_tests)
  1. I converted all the unit tests to pytest (101 tests - it will just skip visualization if graphviz is not installed)
  2. All real LLM tests are into a separate test folder tests/llm_optimizers_tests. Only test_optimizer.py will run for CI for over each optimizer but continue CI on error. I also created a BBH test (should be customized) and OptoPrimeMulti.
  3. Please CHECK OptoPrime modification: tests on different LLM highlighted a frequent confusions between #Variables, #Input and #Code for trainable params; I propose a new version of construct_update_dict to better fallback, you might disagree
  4. LLM class properly handle AutoGen / LiteLLM / CustomLLM
  5. I inserted a section into README.MD to explain a little bit (I will add more following our exchanges)
  6. Minor fixes (TextGrad LiteLLM issue, setup.py)

doxav added 4 commits May 20, 2025 05:12
Uses Phi4-mini ollama run for real LLM tests (moved tests requiring real LLM out of unit_tests)
LLM class properly handle AutoGen / LiteLLM / CustomLLM
Minor fixes
OptoPrimeMulti add async for multi candidates technique which are compatible
@chinganc
Copy link
Member

@doxav Thanks for contributing this PR!! I was just about to do the pytest migration thing too.

Copy link
Member

@chinganc chinganc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@doxav Mostly look good to me. Can you separate the changes to optoprime.py into a separate PR? I think the rest should be mergeable once I do some tests. We're planning to do an overhaul of optoprime and it will be cleaner to have a separate thread on that. Thanks!

@chinganc chinganc merged commit 4056262 into AgentOpt:main May 30, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants