mlflow

Track machine-learning experiments, parameters, metrics, artifacts, and model-registry changes with MLflow.

SKILL.md

GitHub

MLflow Experiment Tracking AI Skill Guide

Overview & Engine Architecture

MLflow records experiments as runs with parameters, metrics, tags, and artifacts. A tracking server (or local mlruns/) stores metadata; the Model Registry promotes versions through stages/aliases. Agents log enough to reproduce a run (code version, data URI, params), avoid silent metric overwrites, and register only evaluated candidates.

Training code
   -> mlflow.start_run
   -> log_param / log_metric / log_artifact
   -> log_model -> Registry (aliases/stages)

When to use this skill

  • Comparing hyperparameters and model families
  • Attaching plots, confusion matrices, and conda/pip envs to runs
  • Promoting models for batch/online inference with audit trail

Operational directives

  1. Set experiment_name explicitly; do not dump everything into Default.
  2. Log data version / query / path as params or tags.
  3. Use nested runs for sweeps when parent/child clarity helps.
  4. Prefer log_model with signature/input example for serving readiness.
  5. Gate registry promotion on held-out metrics, not training loss alone.

Tracking example

import mlflow
import mlflow.sklearn
from mlflow.models import infer_signature

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("churn-models")

with mlflow.start_run(run_name="hgb-v3") as run:
    mlflow.log_params({"max_depth": 6, "learning_rate": 0.08})
    # ... train sklearn pipeline `clf` ...
    mlflow.log_metrics({"roc_auc": 0.91, "f1": 0.74})
    sig = infer_signature(X_train, clf.predict(X_train))
    mlflow.sklearn.log_model(clf, artifact_path="model", signature=sig)
    mlflow.set_tags({"data": "s3://bucket/churn_v4.parquet", "git_sha": "abc123"})

CLI

mlflow ui --backend-store-uri sqlite:///mlflow.db --port 5000
mlflow runs list --experiment-name churn-models

Common failures

Symptom Cause Fix
Empty UI wrong tracking URI align train + UI URIs
Cannot compare inconsistent metric names standardize naming
Huge artifacts logging full datasets log hashes/paths instead
Registry clutter auto-register every run register only candidates

Best practices

  • One primary metric per experiment for sorting.
  • Log environment (mlflow.sklearn.autolog carefully - still review).
  • Immutable training data snapshots or lake table versions.
  • Pair with @scikit-learn / @pytorch / @huggingface-transformers.

Limitations

  • Auth, multi-tenant servers, and artifact stores need ops setup.
  • Autolog can miss custom loops - log manually when needed.
  • Model serving deployment is separate from tracking.

Related skills

  • @scikit-learn / @pytorch - training frameworks
  • @airflow / @prefect - schedule training jobs that log to MLflow
  • @jupyter - exploratory runs still should log when kept