If you cannot say which code, data and settings produced a model, you cannot reproduce it or trust it. This page explains experiment tracking from the ground up, how MLflow records runs, and how tracking connects training to a model registry and production.
Machine learning work is a long series of trials: different features, hyperparameters, data versions and code changes. Experiment tracking is the practice, and the tooling, for recording each trial automatically so it can be compared, reproduced and audited later. The unit of record is a run: one execution of training code. A run stores parameters (inputs you chose, such as learning rate), metrics (numbers you measured, possibly per step or epoch), artifacts (files: the trained model, plots, a sample of predictions) and metadata (git commit, user, start time, environment). Runs are grouped into experiments, usually one per modelling question.
MLflow, an open-source project started by Databricks, is the most widely used tool and the one used in the examples here; Weights & Biases, Neptune, Comet and cloud-native options (SageMaker Experiments, Vertex AI Experiments) offer the same core ideas.
Without tracking, results live in notebook output cells, spreadsheets and memory. Months later nobody can say which data and settings produced the model in production, or whether a new result is actually better than one from last quarter. Tracking solves several concrete problems:
Tracking is also valuable for work that is not classic model training. Teams evaluating large language model prompts, retrieval settings or fine-tuning runs log each configuration and its evaluation scores in exactly the same way, because the questions are the same: what changed, did it help, and can we get back to the previous version if it did not?
A tracking system has three parts. A client library in your training code sends calls such as log_param and log_metric. A backend store (a database such as Postgres, or local files) holds run metadata, params and metrics. An artifact store (object storage such as S3, or a local directory) holds large files. A web UI reads both to list, filter and chart runs.
Many frameworks support autologging: mlflow.autolog() patches scikit-learn, XGBoost, PyTorch Lightning and others to record parameters, metrics and the model without explicit calls. The tracker connects to the rest of the lifecycle through a model registry: a chosen run's model is registered as a numbered version and given an alias such as champion, which serving code loads by name. That link from deployed model back to run is what makes the whole chain auditable.
# pip install mlflow scikit-learn
import mlflow, mlflow.sklearn
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.25, random_state=42)
mlflow.set_tracking_uri("file:./mlruns") # or http://your-mlflow-server:5000
mlflow.set_experiment("rf-depth-sweep")
for depth in [2, 4, 8, 16]:
with mlflow.start_run(run_name=f"depth={depth}"):
params = {"n_estimators": 200, "max_depth": depth, "random_state": 42}
mlflow.log_params(params)
mlflow.set_tag("dataset", "sklearn-breast-cancer")
model = RandomForestClassifier(**params).fit(X_tr, y_tr)
auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
mlflow.log_metric("val_auc", auc)
mlflow.sklearn.log_model(model, "model") # stored as a run artifact
best = mlflow.search_runs(order_by=["metrics.val_auc DESC"]).iloc[0]
print(best["run_id"], best["params.max_depth"], best["metrics.val_auc"])
# Browse with: mlflow ui (then open http://127.0.0.1:5000)The script runs four training jobs that differ only in max_depth, each inside start_run(). Each logs its parameters, a tag, the validation AUC and the model itself. search_runs returns a pandas DataFrame of all runs in the experiment, so selecting the best is a sort. Running mlflow ui opens a browser view where the runs can be compared side by side, as sketched in the chart (values there are illustrative). Note what makes this reproducible: fixed random_state, all parameters logged, and the model stored with the run. In a real project you would also log the git commit (MLflow does this automatically when run from a git repository via mlflow run or when source info is available) and a reference to the exact data version.
What to log, as a checklist. A good default for every training run is:
requirements.txt or conda file as an artifact).Logging is cheap compared with the cost of a model whose origin is unknown. Hyperparameter search tools such as Optuna or Ray Tune can log each trial as a nested run under a parent, keeping large sweeps organised in one place.
| Tool | Hosting | Strengths | Considerations |
|---|---|---|---|
| MLflow | Self-hosted open source, or managed (Databricks, others) | Open, framework-agnostic, model registry included | UI simpler than commercial tools; you run the server |
| Weights & Biases | SaaS or self-hosted enterprise | Rich dashboards, sweeps, collaboration | Commercial licence for teams |
| Neptune / Comet | SaaS | Metadata querying, comparison views | Commercial |
| SageMaker / Vertex AI Experiments | Cloud-native | Integrated with the cloud ML platform | Tied to one cloud |
| Spreadsheet or notebook notes | None | Zero setup | Not reproducible, does not scale |
./mlruns on a laptop is lost with the laptop; use a shared server for team work.