mlops-pro

Overview

MLOps is DevOps for machine learning: the practices that take a model from a notebook to a reliable production system — experiment tracking, reproducible training, model registries, deployment strategies, and monitoring for data/model drift. Most ML projects fail not on modeling but on everything around it: untracked experiments, unreproducible training, and silent degradation in production.

This skill covers the MLOps lifecycle: tracking experiments, versioning data and models, building training pipelines, deploying safely, and monitoring what matters once live.

When to use

• Setting up experiment tracking (MLflow, Weights & Biases).
• Versioning datasets and models (DVC, model registry).
• Building reproducible training pipelines.
• Deploying models (batch, real-time, edge).
• Monitoring models in production (drift, performance, data quality).
• Choosing MLOps tooling for a team.

Core concepts

• Experiment tracking. Log params, metrics, code version, and artifacts for every run (MLflow, W&B). The question "which run produced this model?" must always have an answer. Track everything; storage is cheap, reruns aren't.
• Reproducibility. Pinned dependencies, seeded randomness, versioned data, containerized training. A training run that can't be reproduced is a rumor. Docker + lockfiles + data versions = reproducibility.
• Data versioning. DVC (git for data, pointer files in git), LakeFS, or dataset snapshots — models are functions of their training data; unversioned data means unexplainable models.
• Model registry. Staging → production lifecycle (MLflow Registry, SageMaker, Vertex): versioned models with metadata, approval gates, and lineage (which data + code + run produced this artifact). Promotion is a deliberate act, not a file copy.
• Feature stores. Feast, Tecton — consistent features between training and serving (the training-serving skew killer), feature reuse across models, point-in-time correctness. Worth it when multiple models share features; overkill for one model.
• Training pipelines. Orchestrated DAGs (Kubeflow, SageMaker Pipelines, Airflow, Metaflow): data validation → preprocessing → training → evaluation → registration. Scheduled retraining with the same pipeline that produced the original model.
• Evaluation gates. Automated checks before promotion: metric thresholds on holdout data, slice evaluations (performance per segment — aggregate metrics hide failures on minorities), comparison against the current production model (champion/challenger).
• Deployment patterns. Batch (scheduled scoring — simplest, cheapest), real-time endpoints (SageMaker, Triton, custom FastAPI), edge (ONNX/TFLite), shadow mode (new model scores live traffic without serving), canary (gradual traffic shift with metric comparison).
• Drift monitoring. Data drift (input distribution shifting), concept drift (relationships changing), prediction drift (output distribution shifting) — Evidently, WhyLabs, or custom statistical tests. Alert on drift, investigate before retraining blindly.
• Data quality checks. Schema validation, null rates, range checks on serving inputs (Great Expectations, custom) — most "model degradations" are data pipeline breakages wearing a costume.
• Feedback loops. Capture ground truth when it arrives (delayed labels are normal); close the loop for retraining. Without labels, you're flying blind on actual performance.
• Cost management. GPU utilization (idle GPUs are money burning), spot/preemptible training with checkpointing, right-sizing inference (smaller models, quantization, batching). Track $/1k predictions like any unit cost.
• Governance. Model cards (intended use, limitations, evaluation), audit trails (who promoted what when), bias/fairness evaluations per slice. Increasingly a regulatory requirement, always a trust requirement.
• A/B testing models. Champion/challenger as a randomized experiment — traffic split with business-metric comparison; the rigorous promotion path beyond offline metrics.
• Explainability. SHAP values and feature importance for debugging ("why this prediction?") and stakeholder trust — global behavior and individual decisions both need answers.

Practical workflow

1. Track from run one. MLflow or W&B logging params/metrics/artifacts in the training script — before the first "real" experiment:

   import mlflow
   with mlflow.start_run():
       mlflow.log_params({"lr": 1e-3, "batch": 64, "arch": "resnet50"})
       # ... train ...
       mlflow.log_metrics({"val_acc": 0.94, "val_loss": 0.21})
       mlflow.pytorch.log_model(model, "model")
   

2. Version data with code. DVC pointer files in git; dataset versions referenced in experiment metadata. Never train on "the CSV on my laptop."
3. Build the pipeline. Orchestrate data → train → evaluate → register as a DAG; the same pipeline runs for initial training and retraining. Containerize each step.
4. Gate promotion. Automated evaluation: threshold checks, slice metrics, champion comparison. Human approval for production promotion — the registry records who/when/why.
5. Deploy with a safety pattern. Shadow first (validate on live traffic), then canary (5% → 50% → 100% with metric comparison), with instant rollback to the previous registry version.
6. Monitor three layers. Data quality (schema, nulls, ranges) → drift (input/prediction distributions) → business/performance metrics (with delayed labels). Alert on the first, investigate the second, optimize the third.

   # pseudocode: drift check on serving inputs vs training baseline
   report = evidently.report(reference=training_df, current=serving_window_df)
   if report.drift_detected("income", threshold=0.1):
       alert("data drift: income distribution shifted")
   

7. Close the feedback loop. Capture labels/outcomes; scheduled retraining jobs using the same pipeline; compare challenger vs champion before promoting.
8. Document and govern. Model cards per production model; audit trail in the registry; slice evaluations for fairness; cost dashboards per model.

Common pitfalls

• No experiment tracking — "which run was best?" unanswerable; track from day one.
• Unversioned training data — unreproducible, unexplainable models; DVC/LakeFS.
• Training-serving skew — different preprocessing in training vs serving; feature store or shared code.
• Aggregate metrics only — failing silently on segments; slice evaluations always.
• Deploying without shadow/canary — production as the test environment; gradual rollouts.
• Monitoring predictions, not data — drift detected late; data quality checks first.
• Retraining blindly on drift — drift from a broken pipeline, not a changed world; investigate first.
• No feedback loop — never learning actual performance; capture ground truth.
• GPU waste — idle/oversized instances; utilization monitoring, spot training, right-sizing.
• Notebook-to-production copy-paste — untested, unreviewed code serving traffic; pipelines with CI.
• No rollback plan — bad model stuck in production; registry versions + instant rollback.
• Ignoring data quality — "model degradation" that's actually upstream breakage; validate inputs.
• Skipping model cards/governance — unknown limitations and no audit trail; document intended use and limits.
• No champion/challenger discipline — promoting on offline metrics alone; online experiments measuring business impact before full rollout.