Model Versioning & Reproducibility Using MLflow + Git LFS

Industry Applications

ISO/RTO dispatch forecasting, utility-scale solar/wind curtailment modeling, FERC-mandated market participation

Key Standards

ISO 61850-8-10 (IEC), FERC Part 35, NIST SP 800-204B, CAISO Rule 28

Typical Scale

200–400 MLflow registered models/year per ISO region; 12–38 TB Git LFS storage/year

⚠️ Why It Matters

1

Unversioned model binaries

↓

2

Inability to reproduce ISO dispatch forecasts

↓

3

Failed audit during NERC/NERC-FERC compliance review

↓

4

Operational rollback delay >4 hours

↓

5

Regulatory penalty or market participation suspension

📘 Definition

Model versioning is the systematic tracking, labeling, and storage of machine learning model artifacts—including parameters, hyperparameters, training data snapshots, metrics, and environment specifications—to ensure deterministic reproduction of inference or training outcomes. When integrated with Git Large File Storage (LFS), it enables atomic, auditable, and scalable version control for binary model assets (e.g., serialized PyTorch models, ONNX graphs) while preserving lineage with source code commits. This forms the foundation of MLOps traceability required for regulatory compliance, root-cause analysis, and ISO-compliant dispatch integration in energy forecasting systems.

🎨 Concept Diagram

AI-generated illustration for visual understanding

💡 Engineering Insight

Never treat model versioning as a post-hoc logging exercise—design the Git LFS filter rules and MLflow tracking URI *before* sensor fusion architecture is finalized. In one ERCOT project, a misconfigured .gitattributes rule caused 87% of sky camera embeddings to bypass LFS, bloating the repo to 42 GB and breaking CI timeouts; the fix required re-ingesting 14 months of imagery with content-addressed object keys aligned to ISO dispatch cycle boundaries.

📖 Detailed Explanation

At its core, model versioning ensures that every forecast output can be traced to a specific combination of code, data, and environment—just as civil engineers trace structural loads to specific material test reports and drawing revisions. Git LFS extends Git’s versioning capability to large binary artifacts (e.g., 1.2 GB XGBoost models trained on fused NWP+SCADA time series) by replacing file contents with lightweight text pointers, while storing actual binaries in remote object storage.

The engineering rigor comes from coupling Git LFS with MLflow’s native experiment/run/model registry hierarchy: each ISO dispatch cycle maps to a Git branch; each training run maps to a commit-sha–tagged MLflow run; and each validated model registers with semantic versioning (e.g., v2.3.1-iso61850-q32024). Crucially, MLflow’s model signature enforces input/output schema contracts—preventing accidental deployment of a model expecting 5-minute SCADA intervals to a 1-minute pipeline.

Advanced deployments enforce cross-system consistency: MLflow model URIs are embedded into IEC 61850 GOOSE messages for real-time dispatch handoff; Git LFS object hashes are signed using hardware security modules (HSMs) to satisfy FERC cybersecurity requirements; and uncertainty quantiles are validated against NIST SP 800-204B digital twin verification protocols. This transforms versioning from a developer convenience into an auditable, standards-compliant engineering control layer.

🔄 Engineering Workflow

Step 1

Step 1: Tag Git commit with ISO dispatch cycle ID (e.g., 'CAISO-2024Q3-Dispatch-087')

→

Step 2

Step 2: Capture raw sensor streams (sky camera frames, SCADA timestamps, NWP netCDF headers) as immutable Databricks Delta tables with versioned storage paths

→

Step 3

Step 3: Launch MLflow training run with explicit git_sha, experiment_id, and artifact_root pointing to S3://forecast-models/{dispatch_cycle}/

→

Step 4

Step 4: Register model in MLflow Registry with stage='Staging', constraints='ISO-61850-8-10 compliant uncertainty quantification'

→

Step 5

Step 5: Automate CI/CD promotion only after passing CAISO-specified backtest suite (NREL PVWatts + ERCOT historical load correlation ≥0.92)

→

Step 6

Step 6: Deploy to Kubernetes inference service with sidecar logging of input/output tensors + SHA256 checksums

→

Step 7

Step 7: Archive run metadata (including GPU device UUID and CUDA version) to air-gapped audit vault per FERC Part 35

📋 Decision Guide

Rock/Field Condition	Recommended Design Action
Forecast error spike (>12% MAE) coinciding with NWP model upgrade (e.g., GFS v16 → v17)	Reproduce prior run using exact git commit + pinned NWP API version; compare MLflow run diffs for input feature shifts
SCADA sensor recalibration event logged within ±15 min of model training window	Use MLflow’s ‘run_id’ + ‘input_dataset_version’ tag to retrain against pre-recalibration sensor snapshot stored in Delta Lake
ISO dispatch rejection due to non-reproducible uncertainty bands	Audit MLflow ‘model_signature’ and ‘pip_requirements.txt’; validate quantile regression ensemble seed reproducibility via numpy.random.SeedSequence

📊 Key Properties & Parameters

Artifact Size

12 MB – 2.4 GB

Total byte size of serialized model + dependencies (e.g., scaler, feature encoder, calibration curves)

⚡ Engineering Impact:

Dictates Git LFS pointer strategy, storage tier selection (S3 vs. NAS), and CI/CD pipeline timeout configuration

Version Granularity

1–5 runs per production model release (per ISO dispatch cycle)

Scope of change captured per MLflow run: e.g., hyperparameter sweep, feature engineering update, or sensor calibration shift

⚡ Engineering Impact:

Determines audit trail density and ability to isolate root cause of forecast bias drift across SCADA/NWP/sky-camera inputs

Lineage Fidelity

98.7–100% metadata capture rate (target ≥99.5% for ISO audit readiness)

Completeness of provenance capture: git commit hash, Docker image digest, hardware ID (GPU/CPU), and raw sensor timestamp bounds

⚡ Engineering Impact:

Directly enables forensic replay of forecast generation under identical hardware/software conditions—critical for CAISO Form 377 dispute resolution

Registry Promotion Latency

2.1–18.4 minutes (median 4.3 min under ISO-mandated SLA ≤30 min)

Time elapsed between model validation completion and promotion to 'Staging' or 'Production' stage in MLflow Model Registry

⚡ Engineering Impact:

Impacts dispatch schedule adherence; latency >25 min risks missing 15-min ahead ISO gate closure windows

📐 Key Formulas

Lineage Fidelity Ratio

LF = (N_metadata_fields_captured / N_required_by_ISO_audit_checklist) × 100%

Quantifies completeness of reproducibility metadata capture per MLflow run

Variables:

Symbol	Name	Unit	Description
LF	Lineage Fidelity Ratio	%	Quantifies completeness of reproducibility metadata capture per MLflow run
N_metadata_fields_captured	Number of metadata fields captured	dimensionless	Count of reproducibility metadata fields actually recorded in an MLflow run
N_required_by_ISO_audit_checklist	Number of metadata fields required by ISO audit checklist	dimensionless	Count of reproducibility metadata fields mandated by the applicable ISO audit checklist

Typical Ranges:

CAISO Dispatch Cycle

98.7–100%

ERCOT Backtest Validation

99.2–100%

⚠️ ≥99.5% required for automated audit pass

🔧 Open Calculator

Git LFS Object Age Threshold

T_age = log₂(N_dispatch_cycles_retained) × 30 days

Recommended maximum age for LFS objects before archival to cold storage (prevents repo bloat)

Variables:

Symbol	Name	Unit	Description
T_age	LFS Object Age Threshold	days	Recommended maximum age for LFS objects before archival to cold storage
N_dispatch_cycles_retained	Number of Dispatch Cycles Retained	dimensionless	Count of recent dispatch cycles for which LFS object metadata is retained