MLflow

You can use the MLflow platform with Kaptain to collect data and manage the lifecycle of your Machine Learning and Artificial Intelligence models and experiments. MLflow supports distributed architectures, where the tracking servers, backend store, and artifact store can reside in remote hosts. When using MLflow with Kaptain, Percona handles the backend store and MinIO handles the artifact storage per default. The tracking server is handled by Kaptain.

With MLflow, you are able to:

• Gather and track data related to experiments for analysis.
• Package ML/AI code and share it with other data scientists or transfer to production environments.
• Manage and deploy models from ML libraries to model serving and inference platforms.
• Manage the full lifecycle of an MLflow model from a centralized store (model versioning, stage transitions, annotations, etc.)

Prerequisites

• You have launched a Jupyter notebook with Kaptain. (Using another notebook would require you to build a custom image for this purpose)

  IMPORTANT: If you decide to use your own custom image or an image not provided by Kaptain to run a distributed training job, create an image following the recommended container image creation steps. Also ensure MLflow has been installed on it.

• Optional: If you are using SDK, you have set up Kaptain SDK for your desired training framework.

Set up MLflow

MLflow is available in every notebook per default. You do not need to download, nor install it.

To use Kaptain’s MLflow tracking server from an image that is not part of Kaptain’s distribution, nor a child-image of one, the image should have MLflow Python SDK installed on it and the environment variable should be set as:

MLFLOW_TRACKING_URI=http://mlflow-tracking-server.kubeflow

Use MLflow

For more information on how to use MLflow Tracking, MLflow Projects, or MLflow Models, refer to the MLflow documentation.

If you want to see an example of how to log SDK metrics in MLflow, refer to the Log and Hyperparameters Metrics to MLFlow tutorial.