🎓 Lesson 9
D5
Git LFS Workflow for Large-Scale Training Datasets
Git LFS is a tool that lets you store huge files—like satellite imagery or 3D geophysical models—in Git repositories without slowing down version control operations.
🎯 Learning Objectives
- ✓ Explain how Git LFS reduces repository bloat and improves clone/fetch performance for >100 MB training datasets
- ✓ Design a Git LFS tracking configuration for multi-modal renewable forecasting assets (e.g., NetCDF time-series, LiDAR point clouds, synthetic SAR rasters)
- ✓ Apply LFS pre-commit hooks to enforce size-based file classification and prevent accidental non-LFS commits of large binaries
- ✓ Analyze LFS pointer integrity and traceability across dataset versions using Git commit history and LFS object IDs
📖 Why This Matters
In AI-augmented renewable forecasting, training datasets routinely exceed 10–50 GB—think hourly 1-km-resolution wind/solar irradiance cubes spanning 20 years, or co-registered InSAR deformation stacks from mining subsidence monitoring. Standard Git breaks under this load: clones stall, CI pipelines timeout, and team collaboration grinds to a halt. Git LFS solves this by keeping your version history lean and auditable—while preserving full dataset fidelity, lineage, and reproducibility. Without it, model governance fails at the data layer.
📘 Core Principles
Git LFS operates on three foundational principles: (1) Pointer substitution—large files are replaced with small ASCII text files containing SHA256 hashes and metadata; (2) Remote object storage—actual binaries reside in an LFS-compliant server (e.g., GitHub, GitLab, or self-hosted Gitea + MinIO); and (3) Transparent workflow integration—'git clone', 'git checkout', and 'git pull' automatically fetch required LFS objects on-demand. Critically, LFS does *not* alter Git’s core semantics: commits still record exact dataset states, enabling reproducible model training across time. However, LFS requires explicit file-type registration via 'git lfs track', strict branch protection for LFS-protected paths, and careful handling of partial clones in CI/CD environments where full dataset retrieval may be impractical.
📐 LFS Storage Efficiency Ratio
This ratio quantifies the bandwidth and storage savings achieved by LFS versus native Git for large binary assets. It compares the total size of LFS pointer files (in Git history) against the cumulative size of original binaries—revealing scalability limits before adopting LFS in production pipelines.
LFS Efficiency Ratio (LER)
LER = (S_native × N_commits) / (S_pointers + S_binaries)Quantifies storage and bandwidth efficiency gain of LFS over native Git for versioned large binaries.
Variables:
| Symbol | Name | Unit | Description |
|---|---|---|---|
| S_native | Average binary file size | bytes | Size of one large file (e.g., NetCDF, LAS, HDF5) |
| N_commits | Number of commits containing the file | dimensionless | How many times the file appears in Git history (e.g., via rewrites or updates) |
| S_pointers | Total pointer file size | bytes | Sum of all LFS pointer files across all commits |
| S_binaries | Total unique binary size | bytes | Storage footprint of distinct LFS objects (deduplicated) |
Typical Ranges:
Geospatial ML datasets (1–50 GB): 50× – 300×
Seismic or InSAR time series (>100 GB): 200× – 1000×
💡 Worked Example
Problem: A solar forecasting project tracks 42 NetCDF files averaging 850 MB each (total raw = 35.7 GB). Each LFS pointer is 128 bytes. The Git history contains 120 commits, with all files tracked from commit #1.
1.
Step 1: Calculate total pointer size = 42 files × 128 bytes × 120 commits = 645,120 bytes ≈ 0.645 MB
2.
Step 2: Calculate raw binary storage if stored natively in Git = 35.7 GB × 120 = 4.284 TB (due to Git's full-object duplication per commit)
3.
Step 3: Compute LER = (Native Git storage) / (LFS storage) = 4.284 TB / (35.7 GB + 0.645 MB) ≈ 119.9× reduction
Answer:
The LFS Efficiency Ratio is ~120×—meaning LFS reduces Git object database growth by two orders of magnitude compared to native Git for this dataset.
🏗️ Real-World Application
At Ørsted’s North Sea offshore wind forecasting hub, engineers versioned 18-months of high-res (100 m) WRF-LES simulation outputs (14.2 TB total) using Git LFS integrated with Azure DevOps Repos and Azure Blob Storage. They tracked '*.nc', '*.zarr', and '*.las' patterns, enforced pre-commit validation via Python hooks checking file size (>10 MB → auto-track), and gated CI training jobs to fetch only LFS objects tagged with 'v2.3.1-training-set'. This enabled deterministic retraining of their XGBoost irradiance nowcast model across 37 global teams—with full provenance, <90-second clone times, and automated LFS object GC after 6-month retention policies.