VLLM Proxy

A proxy for vLLM.

Requirements

• Python 3.12+

Run for development

# Run production server
uvicorn main:app --host 0.0.0.0 --reload

# Run development server
fastapi dev main.py --host 0.0.0.0

Production

Build for production

bash docker/build.sh

Run for production

cd docker
docker compose up -d

Tests

Quick Start

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
pip install -r test-requirements.txt
./run_tests.sh

For detailed testing documentation, see TESTING.md.