Add Mlflow Container example commands

[naglepuff]

Changes

docker compose changes

A new service has been added to docker-compose.yml. The original intent was to use the off-the-shelf mlflow docker image, but that was missing some key dependencies , so a small dockerfile has been added which installs additional python packages for our use case.

The service spins up an instance of Mlfow, which is configured to use postgres as a backend store and minio as the asset store. It is available at http://localhost:5000.

Some additional environment variables and django settings have been added as well to facilitate communication between all the services.

new commands

In order to start working with Mlflow, a few new django commands have been added.

./manage.py setupmlflow

This command uses psycopg2 to create a new postgres database called mlflow, which the Mlflow server uses as a backend store. I have not thoroughly investigated use of the django ORM to help with this. Maybe there's a better way to integrate the backend store with the existing django database. This way we could use django models to manage mlflow objects as well. I'll leave that to further discussion/follow up.

It also creates a bucket in minio that Mlflow can use as an artifact store.

./manage.py makeexperiment

Mlflow groups related runs into "experiments." This command allows you to create a named experiment to group training runs. It seemed a useful thing to have, and a good starting point for working with the Mlflow API. It doesn't have to stick around.

./manage.py examplelog

This allows users to run a toy ML training run grouped with an optional experiment. If no experiment is provided, if uses the "default" experiment, which Mlflow creates automatically. For now, this is pretty much a copy/paste of a random forest regressor training session.

The actual ML happens in a celery task, and uploads some artifacts to Mlflow, which for now can only be viewed by accessing the Mlflow instance at http://localhost:5000.