Week 4 Lab: Checkpointing, Tracking, and Versioning¶

Course: Software Tools and Techniques for AI

Three things go wrong in real projects:

  1. Training runs for 3 hours, the machine crashes, and everything is lost.
  2. You have 40 saved models called model_final_v2_new_FINAL.pt and no idea which one gave your best number.
  3. Your accuracy dropped from 87% to 81% and you cannot tell whether the code changed or the data changed.

Elements of the lab.

  • Part A (Sections 1-5): save your training so it survives a crash, and track it with TensorBoard, MLflow, and Weights & Biases.
  • Part B (Sections 6-9): version your code with Git, your data with DVC, and your trained models with clear names and tags.

Running task: the same movie mood classifier from Week 3, but now written in PyTorch so it trains over epochs. Epochs are what make checkpointing meaningful. The model is small on purpose - it runs on a normal CPU.

1. Setup¶

Before we can checkpoint or track anything, we need the libraries this lab depends on, and the shared model/data we will reuse in every section.

Libraries Used¶

  • PyTorch (torch) - builds and trains the neural network, and saves/loads checkpoints.
  • NumPy / Pandas - numerical operations and tabular data handling.
  • Pathlib - platform-independent file and directory handling.
  • json, time, os, shutil - reading configs, timing runs, and managing files on disk.
In [1]:
# !pip install torch tensorboard mlflow wandb dvc dvc-gdrive scikit-learn pandas
In [2]:
import torch
import torch.nn as nn
import numpy as np
import pandas as pd
import json, time, os, shutil
from pathlib import Path

CKPT_DIR = Path("checkpoints"); CKPT_DIR.mkdir(exist_ok=True)
DATA_DIR = Path("data")

torch.manual_seed(42)
np.random.seed(42)

The model we will use all lab¶

This cell is solved - read it carefully, since you will reuse MoodNet, accuracy(), Xtr/Xval/ytr/yval, and FEATURES in every section below.

In [3]:
import os

for root, dirs, files in os.walk("/kaggle/input"):
    for f in files:
        print(os.path.join(root, f))

/kaggle/input/datasets/sarthaksameerpatil/movies-labelled/movies_labelled.csv
/kaggle/input/pm-126908155-at-08-11-2026-10-48-52/__script__.py
/kaggle/input/pm-126908155-at-08-11-2026-10-48-52/__results__.html
/kaggle/input/pm-126908155-at-08-11-2026-10-48-52/input_requirements.txt
/kaggle/input/pm-126908155-at-08-11-2026-10-48-52/__script__.ipynb
/kaggle/input/pm-126908155-at-08-11-2026-10-48-52/__output__.json
/kaggle/input/pm-126908155-at-08-11-2026-10-48-52/custom.css
In [4]:
# SOLVED - read this, you will use it in every section

import pandas as pd
import torch
import torch.nn as nn

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder

# Load dataset
df = pd.read_csv(
    "/kaggle/input/datasets/sarthaksameerpatil/movies-labelled/movies_labelled.csv"
)

FEATURES = [
    "year",
    "runtime_min",
    "brightness",
    "saturation",
    "n_faces"
]

# Features
X = StandardScaler().fit_transform(
    df[FEATURES].fillna(0).values
).astype("float32")

# Labels
le = LabelEncoder()
y = le.fit_transform(df["mood"]).astype("int64")

# Train-validation split
Xtr, Xval, ytr, yval = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

# Convert to PyTorch tensors
Xtr, Xval = torch.tensor(Xtr), torch.tensor(Xval)
ytr, yval = torch.tensor(ytr), torch.tensor(yval)


class MoodNet(nn.Module):

    def __init__(
        self,
        n_in=len(FEATURES),
        n_hidden=32,
        n_out=len(le.classes_)
    ):
        super().__init__()

        self.net = nn.Sequential(
            nn.Linear(n_in, n_hidden),
            nn.ReLU(),

            nn.Linear(n_hidden, n_hidden),
            nn.ReLU(),

            nn.Linear(n_hidden, n_out)
        )

    def forward(self, x):
        return self.net(x)


def accuracy(model, Xd, yd):
    model.eval()

    with torch.no_grad():
        return (
            (model(Xd).argmax(1) == yd)
            .float()
            .mean()
            .item()
        )

PART A - SAVING AND TRACKING TRAINING¶

2. Model Checkpointing¶

A checkpoint is a saved copy of your training, written to disk during training. If the machine dies at epoch 80, you restart from epoch 80 - not from zero.

A checkpoint must hold more than the weights. It also needs the optimiser state, because optimisers like Adam keep their own running numbers. If you save only the weights, training restarts badly.

Why is this important?¶

Without checkpoints, a crash means

  • lost compute time
  • lost progress
  • a training run that must start over from epoch 0

Learning Outcomes¶

  • Save and load a full training checkpoint (weights + optimiser state).
  • Prove that resuming from a checkpoint continues training smoothly.
  • Understand why the optimiser state matters.
  • Decide how often to checkpoint, and clean up old checkpoints.

Question 1.1 (Solved): A basic training loop, no checkpoints¶

In [5]:
# SOLVED EXAMPLE
def train_plain(epochs=30):
    model = MoodNet()
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    loss_fn = nn.CrossEntropyLoss()
    for ep in range(epochs):
        model.train()
        opt.zero_grad()
        loss = loss_fn(model(Xtr), ytr)
        loss.backward()
        opt.step()
        if ep % 10 == 0:
            print(f"epoch {ep:3d} | loss {loss.item():.4f} | val {accuracy(model, Xval, yval):.4f}")
    return model

model = train_plain()

epoch   0 | loss 1.6071 | val 0.3542
epoch  10 | loss 0.9652 | val 0.6083
epoch  20 | loss 0.7881 | val 0.6708

If the notebook crashed halfway through that loop, everything would be gone.

Question 1.2 (Solved): What goes inside a checkpoint¶

In [6]:
# SOLVED EXAMPLE
def save_checkpoint(path, model, optimizer, epoch, val_acc, extra=None):
    torch.save({
        "epoch": epoch,
        "model_state": model.state_dict(),        # the weights
        "optimizer_state": optimizer.state_dict(),# the optimiser's memory
        "val_acc": val_acc,
        "extra": extra or {},                     # anything else you need
    }, path)

def load_checkpoint(path, model, optimizer=None):
    ckpt = torch.load(path)
    model.load_state_dict(ckpt["model_state"])
    if optimizer is not None:
        optimizer.load_state_dict(ckpt["optimizer_state"])
    return ckpt["epoch"], ckpt["val_acc"]

Question 1.3: Add checkpointing to the loop¶

Rewrite train_plain as train_with_ckpt(epochs=50, every=10). It must save a checkpoint every every epochs to checkpoints/epoch_{ep}.pt.

Print the file size of one checkpoint in kilobytes.

In [7]:
# YOUR CODE HERE

Question 1.4: The crash test¶

This is the important one. Prove your checkpoints actually work.

  1. Train for 25 epochs, saving checkpoints. Note the loss at epoch 24.
  2. Delete the model and optimiser objects (del model, opt) - this is your "crash".
  3. Make a new model and optimiser, load the epoch-20 checkpoint into them, and continue training from epoch 21 to 50.
  4. Compare: does the loss continue smoothly, or does it jump up?
In [8]:
# YOUR CODE HERE

Question 1.5: Why the optimiser state matters¶

Repeat Question 1.4, but this time load only the model weights and leave the optimiser fresh. Plot both loss curves on one graph:

  • resumed with optimiser state,
  • resumed without optimiser state.

Explain in 2-3 sentences what you see. (Hint: Adam keeps running averages of past gradients. Throwing them away means it must relearn them.)

In [9]:
# YOUR CODE HERE
answer_1_5 = "..."

Question 1.6: How often should you save?¶

Saving every epoch is safe but slow and fills the disk. Saving once at the end is fast but risky.

Fill in this table for your own model, using real measurements:

Frequency Disk used after 100 epochs Work lost if it crashes at epoch 99
every epoch
every 10 epochs
only at the end

Then state the rule you would use for a model that takes 6 hours to train.

In [10]:
# YOUR CODE HERE
answer_1_6 = "..."

Question 1.7: Do not fill the disk¶

Write keep_last_n(dir, n=3) that deletes old checkpoints and keeps only the n newest. Run it and show that only 3 files remain.

In [11]:
# YOUR CODE HERE

3. Best Model vs Last Model¶

The model at the last epoch is often not the best one. After a while, training accuracy keeps rising while validation accuracy falls - that is overfitting. The best model was somewhere in the middle.

Why is this important?¶

Shipping the last-epoch model can mean shipping an overfit model. Tracking the best validation score separately protects you from that mistake.

Learning Outcomes¶

  • Save the best model separately from the last model.
  • Visualise the train/validation gap.
  • Implement early stopping.
  • Justify which model to ship to users.

Question 2.1 (Solved): Save the best, not just the last¶

In [12]:
# SOLVED EXAMPLE
def train_track_best(epochs=200):
    model = MoodNet()
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    loss_fn = nn.CrossEntropyLoss()
    best_acc, best_epoch = -1, -1
    history = []

    for ep in range(epochs):
        model.train(); opt.zero_grad()
        loss = loss_fn(model(Xtr), ytr); loss.backward(); opt.step()

        val = accuracy(model, Xval, yval)
        history.append({"epoch": ep, "loss": loss.item(),
                        "train_acc": accuracy(model, Xtr, ytr), "val_acc": val})

        if val > best_acc:                                 # new best -> overwrite
            best_acc, best_epoch = val, ep
            save_checkpoint(CKPT_DIR / "best.pt", model, opt, ep, val)

    save_checkpoint(CKPT_DIR / "last.pt", model, opt, epochs - 1, val)
    print(f"Best val {best_acc:.4f} at epoch {best_epoch}; last val {val:.4f}")
    return pd.DataFrame(history)

hist = train_track_best()

Best val 0.6792 at epoch 44; last val 0.5875

Question 2.2: Show the gap¶

Plot train_acc and val_acc against epoch from hist. Mark the best epoch with a vertical line.

Then report: the best validation accuracy, the last validation accuracy, and the gap between them. How many epochs were wasted after the best point?

In [13]:
# YOUR CODE HERE

Question 2.3: Early stopping¶

Early stopping ends training when validation stops improving, so you do not waste those epochs.

Write train_early_stop(patience=20). Keep a counter of how many epochs have passed with no improvement. When the counter reaches patience, stop.

Report: the epoch it stopped at, the best accuracy, and the time saved compared to the full 200 epochs.

In [14]:
# YOUR CODE HERE

Question 2.4: Choosing patience¶

Run early stopping with patience = 5, 20, and 50. Fill in a table with the stop epoch and best accuracy for each.

Then answer: what goes wrong if patience is too small? What goes wrong if it is too large?

In [15]:
# YOUR CODE HERE
answer_2_4 = "..."

Question 2.5: Which model do you ship?¶

You have three candidates:

  • the model from the last epoch,
  • the model with the best validation accuracy,
  • the model with the lowest training loss.

Which do you give to users, and why? In your answer, say clearly why picking by training loss is a bad idea.

In [16]:
# YOUR ANSWER HERE
answer_2_5 = "..."

Question 2.6: An honest warning about "best"¶

You chose the best model by looking at the validation set many times - once per epoch. So the validation score is now slightly too optimistic.

Explain why in two sentences, and say what extra split you need to report an honest final number.

In [17]:
# YOUR ANSWER HERE
answer_2_6 = "..."

4. TensorBoard¶

Printing numbers works for 30 epochs. For 3,000 epochs you need charts. TensorBoard reads a log folder and draws them live while training runs.

Why is this important?¶

A chart shows trends (overfitting, plateaus, spikes) that a printed number cannot. Comparing runs side by side is also far easier visually than by scanning printed logs.

Learning Outcomes¶

  • Log scalars, the learning rate, and histograms to TensorBoard.
  • Compare several runs on one chart.
  • Inspect a model's computational graph.

Question 3.1 (Solved): Log scalars to TensorBoard¶

In [18]:
# SOLVED EXAMPLE
from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter("tb_logs/run1")
model = MoodNet()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()

for ep in range(100):
    model.train(); opt.zero_grad()
    loss = loss_fn(model(Xtr), ytr); loss.backward(); opt.step()

    writer.add_scalar("loss/train", loss.item(), ep)
    writer.add_scalar("accuracy/train", accuracy(model, Xtr, ytr), ep)
    writer.add_scalar("accuracy/val", accuracy(model, Xval, yval), ep)

writer.close()

Then open it:

tensorboard --logdir tb_logs
# open http://localhost:6006

In a notebook you can instead run: %load_ext tensorboard then %tensorboard --logdir tb_logs.

Question 3.2: Log the learning rate too¶

Use a learning rate scheduler (torch.optim.lr_scheduler.StepLR with step_size=30, gamma=0.5). Log the current learning rate every epoch with writer.add_scalar("lr", opt.param_groups[0]["lr"], ep).

Look at the chart. Describe what happens to the loss at the moment the learning rate drops.

In [19]:
# YOUR CODE HERE

Question 3.3: Compare three runs on one chart¶

Train three models with different learning rates (0.1, 0.01, 0.001). Write each to its own subfolder: tb_logs/lr_0.1, tb_logs/lr_0.01, tb_logs/lr_0.001.

Open TensorBoard once and view all three together. Which learning rate wins? What does the 0.1 curve look like, and what does that tell you?

In [20]:
# YOUR CODE HERE

Question 3.4: Draw the model graph¶

Use writer.add_graph(model, Xtr[:1]) to log the computational graph. Open the GRAPHS tab.

Then answer: how many Linear layers does the graph show, and does that match your MoodNet definition?

In [21]:
# YOUR CODE HERE

Question 3.5: Log a histogram of weights¶

Every 10 epochs, log the weights of the first layer with writer.add_histogram("layer1_weights", model.net[0].weight, ep).

Open the HISTOGRAMS tab. Do the weights keep spreading out, or do they settle? What would it mean if they grew very large?

In [22]:
# YOUR CODE HERE

Question 3.6: The embedding projector (short answer)¶

TensorBoard's embedding projector takes high-dimensional vectors and draws them in 2D or 3D, so you can see which points group together.

Write 2-3 sentences on how you would use it for this project. What would you feed in, and what would you hope to see if your mood labels are good?

In [23]:
# YOUR ANSWER HERE
answer_3_6 = "..."

5. MLflow¶

TensorBoard shows curves inside one run. MLflow is built to compare many runs, and to store the files each run produced (called artifacts).

Why is this important?¶

Once you have run dozens of experiments, you need a searchable record of what settings produced what score, plus the model and files that run generated.

Learning Outcomes¶

  • Track parameters, metrics, and artifacts with MLflow.
  • Compare many runs in the MLflow UI and from code.
  • Load a saved model back from a run and verify it.
  • Explain the purpose of the MLflow model registry.

Question 4.1 (Solved): Track one run with MLflow¶

In [24]:
!pip install mlflow

Collecting mlflow
  Downloading mlflow-3.15.1-py3-none-any.whl.metadata (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.7/49.7 kB 1.5 MB/s eta 0:00:00
Collecting mlflow-skinny==3.15.1 (from mlflow)
  Downloading mlflow_skinny-3.15.1-py3-none-any.whl.metadata (50 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.5/50.5 kB 2.8 MB/s eta 0:00:00
Collecting mlflow-tracing==3.15.1 (from mlflow)
  Downloading mlflow_tracing-3.15.1-py3-none-any.whl.metadata (19 kB)
Collecting Flask-CORS<7 (from mlflow)
  Downloading flask_cors-6.0.5-py3-none-any.whl.metadata (5.4 kB)
Requirement already satisfied: Flask<4 in /usr/local/lib/python3.12/dist-packages (from mlflow) (3.1.3)
Requirement already satisfied: aiohttp<4,>=3.7.0 in /usr/local/lib/python3.12/dist-packages (from mlflow) (3.13.5)
Requirement already satisfied: alembic!=1.10.0,<2 in /usr/local/lib/python3.12/dist-packages (from mlflow) (1.18.4)
Requirement already satisfied: cryptography<50,>=43.0.0 in /usr/local/lib/python3.12/dist-packages (from mlflow) (43.0.3)
Requirement already satisfied: docker<8,>=4.0.0 in /usr/local/lib/python3.12/dist-packages (from mlflow) (7.1.0)
Collecting graphene<4 (from mlflow)
  Downloading graphene-3.4.3-py2.py3-none-any.whl.metadata (6.9 kB)
Collecting gunicorn<27 (from mlflow)
  Downloading gunicorn-26.0.0-py3-none-any.whl.metadata (5.4 kB)
Collecting huey<4,>=2.5.4 (from mlflow)
  Downloading huey-3.3.4-py3-none-any.whl.metadata (5.6 kB)
Requirement already satisfied: matplotlib<4 in /usr/local/lib/python3.12/dist-packages (from mlflow) (3.10.0)
Requirement already satisfied: numpy<3 in /usr/local/lib/python3.12/dist-packages (from mlflow) (2.0.2)
Requirement already satisfied: pandas<3 in /usr/local/lib/python3.12/dist-packages (from mlflow) (2.3.3)
Requirement already satisfied: pyarrow<26,>=4.0.0 in /usr/local/lib/python3.12/dist-packages (from mlflow) (24.0.0)
Requirement already satisfied: scikit-learn<2 in /usr/local/lib/python3.12/dist-packages (from mlflow) (1.6.1)
Requirement already satisfied: scipy<2 in /usr/local/lib/python3.12/dist-packages (from mlflow) (1.16.3)
Collecting skops<1 (from mlflow)
  Downloading skops-0.14.0-py3-none-any.whl.metadata (4.4 kB)
Requirement already satisfied: sqlalchemy<3,>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from mlflow) (2.0.49)
Requirement already satisfied: cachetools<8,>=5.0.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (6.2.6)
Requirement already satisfied: click<9,>=7.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (8.3.3)
Requirement already satisfied: cloudpickle<4 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (3.1.2)
Collecting databricks-sdk<1,>=0.20.0 (from mlflow-skinny==3.15.1->mlflow)
  Downloading databricks_sdk-0.126.0-py3-none-any.whl.metadata (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.0/44.0 kB 2.2 MB/s eta 0:00:00
Requirement already satisfied: fastapi<1 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (0.136.1)
Requirement already satisfied: gitpython<4,>=3.1.9 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (3.1.47)
Requirement already satisfied: importlib_metadata!=4.7.0,<10,>=3.7.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (8.7.1)
Requirement already satisfied: opentelemetry-api<3,>=1.9.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (1.38.0)
Requirement already satisfied: opentelemetry-proto<3,>=1.9.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (1.38.0)
Requirement already satisfied: opentelemetry-sdk<3,>=1.9.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (1.38.0)
Requirement already satisfied: packaging<27 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (26.1)
Requirement already satisfied: protobuf<8,>=3.12.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (5.29.5)
Requirement already satisfied: pydantic<3,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (2.12.3)
Requirement already satisfied: python-dotenv<2,>=0.19.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (1.2.2)
Requirement already satisfied: pyyaml<7,>=5.1 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (6.0.3)
Requirement already satisfied: requests<3,>=2.17.3 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (2.32.4)
Requirement already satisfied: sqlparse<1,>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (0.5.5)
Requirement already satisfied: starlette<2 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (0.52.1)
Requirement already satisfied: typing-extensions<5,>=4.0.0 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (4.15.0)
Requirement already satisfied: uvicorn<1 in /usr/local/lib/python3.12/dist-packages (from mlflow-skinny==3.15.1->mlflow) (0.46.0)
Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4,>=3.7.0->mlflow) (2.6.1)
Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4,>=3.7.0->mlflow) (1.4.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4,>=3.7.0->mlflow) (26.1.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4,>=3.7.0->mlflow) (1.8.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4,>=3.7.0->mlflow) (6.7.1)
Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4,>=3.7.0->mlflow) (0.4.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4,>=3.7.0->mlflow) (1.23.0)
Requirement already satisfied: Mako in /usr/local/lib/python3.12/dist-packages (from alembic!=1.10.0,<2->mlflow) (1.3.11)
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.12/dist-packages (from cryptography<50,>=43.0.0->mlflow) (2.0.0)
Requirement already satisfied: urllib3>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from docker<8,>=4.0.0->mlflow) (2.5.0)
Requirement already satisfied: blinker>=1.9.0 in /usr/local/lib/python3.12/dist-packages (from Flask<4->mlflow) (1.9.0)
Requirement already satisfied: itsdangerous>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from Flask<4->mlflow) (2.2.0)
Requirement already satisfied: jinja2>=3.1.2 in /usr/local/lib/python3.12/dist-packages (from Flask<4->mlflow) (3.1.6)
Requirement already satisfied: markupsafe>=2.1.1 in /usr/local/lib/python3.12/dist-packages (from Flask<4->mlflow) (3.0.3)
Requirement already satisfied: werkzeug>=3.1.0 in /usr/local/lib/python3.12/dist-packages (from Flask<4->mlflow) (3.1.8)
Collecting graphql-core<3.3,>=3.1 (from graphene<4->mlflow)
  Downloading graphql_core-3.2.11-py3-none-any.whl.metadata (11 kB)
Collecting graphql-relay<3.3,>=3.1 (from graphene<4->mlflow)
  Downloading graphql_relay-3.2.0-py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: python-dateutil<3,>=2.7.0 in /usr/local/lib/python3.12/dist-packages (from graphene<4->mlflow) (2.9.0.post0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib<4->mlflow) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib<4->mlflow) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib<4->mlflow) (4.62.1)
Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib<4->mlflow) (1.5.0)
Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib<4->mlflow) (11.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib<4->mlflow) (3.3.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas<3->mlflow) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas<3->mlflow) (2026.1)
Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn<2->mlflow) (1.5.3)
Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn<2->mlflow) (3.6.0)
Requirement already satisfied: prettytable>=3.9 in /usr/local/lib/python3.12/dist-packages (from skops<1->mlflow) (3.17.0)
Requirement already satisfied: greenlet>=1 in /usr/local/lib/python3.12/dist-packages (from sqlalchemy<3,>=1.4.0->mlflow) (3.4.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.12/dist-packages (from cffi>=1.12->cryptography<50,>=43.0.0->mlflow) (3.0)
Requirement already satisfied: google-auth~=2.0 in /usr/local/lib/python3.12/dist-packages (from databricks-sdk<1,>=0.20.0->mlflow-skinny==3.15.1->mlflow) (2.47.0)
Requirement already satisfied: typing-inspection>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from fastapi<1->mlflow-skinny==3.15.1->mlflow) (0.4.2)
Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from fastapi<1->mlflow-skinny==3.15.1->mlflow) (0.0.4)
Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.12/dist-packages (from gitpython<4,>=3.1.9->mlflow-skinny==3.15.1->mlflow) (4.0.12)
Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.12/dist-packages (from importlib_metadata!=4.7.0,<10,>=3.7.0->mlflow-skinny==3.15.1->mlflow) (3.23.1)
Requirement already satisfied: opentelemetry-semantic-conventions==0.59b0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-sdk<3,>=1.9.0->mlflow-skinny==3.15.1->mlflow) (0.59b0)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.12/dist-packages (from prettytable>=3.9->skops<1->mlflow) (0.6.0)
Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic<3,>=2.0.0->mlflow-skinny==3.15.1->mlflow) (0.7.0)
Requirement already satisfied: pydantic-core==2.41.4 in /usr/local/lib/python3.12/dist-packages (from pydantic<3,>=2.0.0->mlflow-skinny==3.15.1->mlflow) (2.41.4)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil<3,>=2.7.0->graphene<4->mlflow) (1.17.0)
Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3,>=2.17.3->mlflow-skinny==3.15.1->mlflow) (3.4.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests<3,>=2.17.3->mlflow-skinny==3.15.1->mlflow) (3.13)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests<3,>=2.17.3->mlflow-skinny==3.15.1->mlflow) (2026.4.22)
Requirement already satisfied: anyio<5,>=3.6.2 in /usr/local/lib/python3.12/dist-packages (from starlette<2->mlflow-skinny==3.15.1->mlflow) (4.13.0)
Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.12/dist-packages (from uvicorn<1->mlflow-skinny==3.15.1->mlflow) (0.16.0)
Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.12/dist-packages (from gitdb<5,>=4.0.1->gitpython<4,>=3.1.9->mlflow-skinny==3.15.1->mlflow) (5.0.3)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.12/dist-packages (from google-auth~=2.0->databricks-sdk<1,>=0.20.0->mlflow-skinny==3.15.1->mlflow) (0.4.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.12/dist-packages (from google-auth~=2.0->databricks-sdk<1,>=0.20.0->mlflow-skinny==3.15.1->mlflow) (4.9.1)
Requirement already satisfied: pyasn1<0.7.0,>=0.6.1 in /usr/local/lib/python3.12/dist-packages (from pyasn1-modules>=0.2.1->google-auth~=2.0->databricks-sdk<1,>=0.20.0->mlflow-skinny==3.15.1->mlflow) (0.6.3)
Downloading mlflow-3.15.1-py3-none-any.whl (11.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/11.2 MB 79.1 MB/s eta 0:00:00
Downloading mlflow_skinny-3.15.1-py3-none-any.whl (3.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 84.6 MB/s eta 0:00:00
Downloading mlflow_tracing-3.15.1-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 46.2 MB/s eta 0:00:00
Downloading flask_cors-6.0.5-py3-none-any.whl (16 kB)
Downloading graphene-3.4.3-py2.py3-none-any.whl (114 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.9/114.9 kB 6.7 MB/s eta 0:00:00
Downloading gunicorn-26.0.0-py3-none-any.whl (212 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.0/212.0 kB 13.2 MB/s eta 0:00:00
Downloading huey-3.3.4-py3-none-any.whl (123 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.9/123.9 kB 7.4 MB/s eta 0:00:00
Downloading skops-0.14.0-py3-none-any.whl (132 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.2/132.2 kB 6.4 MB/s eta 0:00:00
Downloading databricks_sdk-0.126.0-py3-none-any.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 52.3 MB/s eta 0:00:00
Downloading graphql_core-3.2.11-py3-none-any.whl (214 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 214.9/214.9 kB 12.1 MB/s eta 0:00:00
Downloading graphql_relay-3.2.0-py3-none-any.whl (16 kB)
Installing collected packages: huey, gunicorn, graphql-core, graphql-relay, skops, graphene, Flask-CORS, databricks-sdk, mlflow-tracing, mlflow-skinny, mlflow
Successfully installed Flask-CORS-6.0.5 databricks-sdk-0.126.0 graphene-3.4.3 graphql-core-3.2.11 graphql-relay-3.2.0 gunicorn-26.0.0 huey-3.3.4 mlflow-3.15.1 mlflow-skinny-3.15.1 mlflow-tracing-3.15.1 skops-0.14.0
In [25]:
import mlflow
import mlflow.pytorch

mlflow.set_experiment("movie-mood")

with mlflow.start_run(run_name="mlp-lr0.01"):

    mlflow.log_params({
        "lr": 0.01,
        "hidden": 32,
        "epochs": 100,
        "seed": 42
    })

    model = MoodNet()

    opt = torch.optim.Adam(
        model.parameters(),
        lr=0.01
    )

    loss_fn = nn.CrossEntropyLoss()

    for ep in range(100):

        model.train()
        opt.zero_grad()

        loss = loss_fn(model(Xtr), ytr)

        loss.backward()
        opt.step()

        mlflow.log_metric(
            "train_loss",
            loss.item(),
            step=ep
        )

        mlflow.log_metric(
            "val_acc",
            accuracy(model, Xval, yval),
            step=ep
        )

    # Save PyTorch model using pickle format
    mlflow.pytorch.log_model(
        model,
        name="model",
        input_example=Xtr[:1],
        serialization_format="pickle"
    )

print("Experiment completed successfully!")

2026/08/11 10:50:52 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/08/11 10:50:52 INFO mlflow.store.db.utils: Updating database tables
2026/08/11 10:50:54 INFO mlflow.tracking.fluent: Experiment with name 'movie-mood' does not exist. Creating a new experiment.
2026/08/11 10:50:58 WARNING mlflow.pytorch: Saving pytorch model by Pickle or CloudPickle format requires exercising caution because these formats rely on Python's object serialization mechanism, which can execute arbitrary code during deserialization. The recommended safe alternative is to set `serialization_format` to 'pt2' to save the PyTorch model using the safe graph model format.
2026/08/11 10:50:58 WARNING mlflow.utils.requirements_utils: Found torch version (2.10.0+cpu) contains a local version label (+cpu). MLflow logged a pip requirement for this package as 'torch==2.10.0' without the local version label to make it installable from PyPI. To specify pip requirements containing local version labels, please use `conda_env` or `pip_requirements`.
2026/08/11 10:51:15 WARNING mlflow.utils.requirements_utils: Found torchvision version (0.25.0+cpu) contains a local version label (+cpu). MLflow logged a pip requirement for this package as 'torchvision==0.25.0' without the local version label to make it installable from PyPI. To specify pip requirements containing local version labels, please use `conda_env` or `pip_requirements`.

Experiment completed successfully!
In [26]:
import mlflow

print("Tracking URI:", mlflow.get_tracking_uri())

Tracking URI: sqlite:////kaggle/working/mlflow.db

Open the UI:

mlflow ui
# open http://localhost:5000

Question 4.2: Log six runs and compare them¶

Run the same training with six different settings (vary lr and hidden). Give each run a clear name.

In the MLflow UI, sort by val_acc and take a screenshot of the table. Which setting won?

In [27]:
# YOUR CODE HERE

Question 4.3: Save useful artifacts¶

Add three artifacts to one run:

  1. The confusion matrix as a PNG image.
  2. The history DataFrame as a CSV.
  3. Your requirements.txt from Week 3.

Show that you can download all three from the MLflow UI.

In [28]:
# YOUR CODE HERE

Question 4.4: Search your runs from code¶

Use mlflow.search_runs() to load all your runs into a DataFrame. Then write code that finds the run with the highest val_acc and prints its parameters.

In [29]:
# YOUR CODE HERE

Question 4.5: Load a model back and use it¶

Take the run id of your best run. Load the model with mlflow.pytorch.load_model(f"runs:/{run_id}/model") and use it to predict on Xval. Check that the accuracy matches what was logged.

If it does not match, that is a bug worth finding - say what could cause it.

In [30]:
# YOUR CODE HERE

Question 4.6: The model registry (short answer)¶

MLflow has a model registry: a named place where models get versions (v1, v2) and stages (Staging, Production, Archived).

Explain in 3-4 sentences why a team needs this, when logging models to runs already works. Think about: which model is live right now, and how you would go back if a new one turns out worse.

In [31]:
# YOUR ANSWER HERE
answer_4_6 = "..."

6. Weights & Biases¶

You used W&B for sweeps last week. Now use it for checkpoints and comparison.

Why is this important?¶

W&B stores checkpoints as versioned artifacts in the cloud, so a teammate can pull the exact model that produced a given score without you sending files by hand.

Learning Outcomes¶

  • Log training curves and a checkpoint artifact to W&B.
  • Compare multiple runs in the W&B dashboard.
  • Download an artifact back and verify it.
  • Choose the right experiment-tracking tool for a given situation.

Question 5.1 (Solved): Log training and a checkpoint to W&B¶

In [32]:
import wandb
print(wandb.__version__)

0.26.1
In [ ]:
import wandb

wandb.init(
    project="movie-mood-week4",
    name="mlp-baseline",
    config={
        "lr": 0.01,
        "hidden": 32,
        "epochs": 100
    }
)

model = MoodNet()

opt = torch.optim.Adam(
    model.parameters(),
    lr=wandb.config.lr
)

loss_fn = nn.CrossEntropyLoss()

best = -1

for ep in range(wandb.config.epochs):

    model.train()
    opt.zero_grad()

    loss = loss_fn(model(Xtr), ytr)

    loss.backward()
    opt.step()

    val = accuracy(model, Xval, yval)

    wandb.log(
        {
            "train_loss": loss.item(),
            "val_acc": val
        },
        step=ep
    )

    if val > best:
        best = val

        save_checkpoint(
            CKPT_DIR / "wandb_best.pt",
            model,
            opt,
            ep,
            val
        )

# Upload checkpoint as a versioned W&B artifact
art = wandb.Artifact(
    "mood-model",
    type="model"
)

art.add_file(
    str(CKPT_DIR / "wandb_best.pt")
)

wandb.log_artifact(art)

wandb.finish()

print("W&B experiment completed!")
print("Best validation accuracy:", best)
print("Checkpoint:", CKPT_DIR / "wandb_best.pt")

Question 5.2: Compare runs in the dashboard¶

Run the training three times with different hidden sizes (16, 32, 64). In the W&B project page, put all three on one chart.

Report which won, and paste the link to your project.

In [ ]:
# YOUR CODE HERE

Question 5.3: Download an artifact back¶

Use wandb.use_artifact("mood-model:latest") and .download() to fetch your checkpoint from the cloud. Load it and confirm the accuracy.

Then answer: why is downloading a model by its artifact name and version safer than emailing a .pt file to a teammate?

In [ ]:
# YOUR CODE HERE
answer_5_3 = "..."

Question 5.4: Log a confusion matrix and a table¶

Use wandb.log({"conf_mat": wandb.plot.confusion_matrix(...)}) to log a confusion matrix, and wandb.Table to log 10 example predictions with the true label and predicted label.

Look at the wrong predictions. Do they match the mood pairs that annotators confused in Week 2?

In [ ]:
# YOUR CODE HERE

Question 5.5: Choose the right tool¶

You have now used TensorBoard, MLflow, and W&B. Fill in this table with your own judgement, one short line per cell:

TensorBoard MLflow W&B
Best at
Works offline?
Good for teams?
Setup effort
Who stores the data

Then pick one tool for this course project and defend the choice in 2 sentences.

In [ ]:
# YOUR ANSWER HERE
answer_5_5 = "..."

PART B - VERSIONING CODE, DATA, AND MODELS¶

7. Git Basics¶

Version control keeps the full history of your code, so you can go back to any past state and see who changed what.

Key words:

Word Meaning
repository (repo) a folder whose history Git tracks
commit a saved snapshot with a message
branch a separate line of work
merge joining a branch back into another
remote a copy on a server, like GitHub

Why is this important?¶

Without version control, "which version of the code produced this number" is unanswerable. Git makes every past state recoverable and every change attributable.

Learning Outcomes¶

  • Initialise a repo and make commits.
  • Branch, change, and merge.
  • Resolve a merge conflict.
  • Push to GitHub and tag a commit.

Note¶

Sections 6-8 are run in a terminal, not in this notebook's Python kernel. Use ! at the start of a notebook cell to run a shell command (e.g. !git status), or open an actual terminal alongside this notebook.

Question 6.1 (Solved): Start a repo and make commits¶

In [ ]:
%cd /kaggle/working

!mkdir -p movie-mood
%cd movie-mood

!git init

!echo "# Movie Mood Classifier" > README.md

!git add README.md

!git commit -m "First commit: add README"

!git log --oneline

!git status

Question 6.2: Build a real history¶

In your project folder:

  1. Add train.py (use your Week 3 script) and commit it.
  2. Change the learning rate, and commit again with a clear message.
  3. Add a .gitignore that ignores checkpoints/, data/, __pycache__/, and *.pt. Commit it.
  4. Run git log --oneline and paste the output.

Then answer: why do we ignore data/ and *.pt instead of committing them?

In [ ]:
# YOUR CODE HERE

Question 6.3: Branch, change, merge¶

  1. Make a branch: git checkout -b try-bigger-model.
  2. On that branch, change n_hidden from 32 to 128 and commit.
  3. Go back to main and check the file - is it changed there? Explain why.
  4. Merge the branch into main and confirm the change arrived.
In [ ]:
# YOUR ANSWER HERE
answer_6_3 = "..."

Question 6.4: Cause a merge conflict on purpose¶

Conflicts scare beginners, so create one deliberately and fix it.

  1. On main, change the learning rate to 0.05 and commit.
  2. On a new branch from before that change, set it to 0.001 and commit.
  3. Merge. Git will report a conflict.
  4. Open the file, read the <<<<<<< markers, choose one version, and commit the fix.

Paste what the conflicted file looked like before you fixed it.

In [ ]:
# YOUR ANSWER HERE
answer_6_4 = "..."

Question 6.5: Push to GitHub¶

Create an empty repo on GitHub, then connect and push:

git remote add origin https://github.com/<you>/movie-mood.git
git branch -M main
git push -u origin main

Submit the link. Then answer: what is the difference between commit and push?

In [ ]:
# YOUR ANSWER HERE
answer_6_5 = "..."

Question 6.6: Tag your experiments¶

A tag marks one commit with a name, so you can find it later.

Tag the commit that produced your best Week 3 result:

git tag -a v1.0-best-rf -m "Best random forest, val acc 0.87"
git push origin v1.0-best-rf

Then explain in two sentences how a tag helps when a professor asks about a result from three months ago.

In [ ]:
# YOUR CODE HERE

8. Dataset Versioning¶

Git is built for text files. It is bad at large binary files. But your data changes too - you add movies, fix labels, drop bad rows - and each change can move your accuracy.

If you cannot say which version of the data produced a number, the number cannot be reproduced.

Why is this important?¶

An accuracy number is meaningless without knowing exactly which data produced it. A fingerprint (hash) of the data file is a cheap, reliable way to say "this is the exact data used".

Learning Outcomes¶

  • Fingerprint a dataset with a hash.
  • Show that even a tiny data change is detected.
  • Measure how a data change affects model accuracy.
  • Explain why Git alone cannot version large datasets well.

Question 7.1 (Solved): Fingerprint a dataset¶

In [ ]:
import hashlib

def file_hash(path):
    """A short fingerprint of a file. Same content -> same hash."""
    h = hashlib.md5()

    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)

    return h.hexdigest()[:12]


path = "/kaggle/input/datasets/sarthaksameerpatil/movies-labelled/movies_labelled.csv"

print("Data version:", file_hash(path))

Question 7.2: Prove that a small change is detected¶

  1. Print the hash of your CSV.
  2. Change one single label in the file and save it.
  3. Print the hash again.

Show the two hashes side by side. Then answer: how many characters of the file changed, and how much of the hash changed?

In [ ]:
# YOUR CODE HERE

Question 7.3: Show the data changing your score¶

Train the same model, with the same seed, on two versions of the data:

  • version A: your full dataset,
  • version B: the same data with 10% of the rows removed.

Report both accuracies and both data hashes. This is the exact situation that confuses people who only version their code.

In [ ]:
# YOUR CODE HERE

Question 7.4: Why Git alone is not enough¶

Suppose your dataset is a 4 GB folder of posters, and you change 5 images per week for 30 weeks.

Answer in 3-4 sentences: what happens to the size of the Git repo, and why is that a problem for your teammates who clone it?

In [ ]:
# YOUR ANSWER HERE
answer_7_4 = "..."

9. DVC (Data Version Control)¶

DVC solves the problem above with a neat trick: the big file goes to storage, and only a small pointer file goes into Git.

The pointer file holds the hash. So git checkout of an old commit, followed by dvc checkout, brings back exactly the data that commit used.

Why is this important?¶

DVC lets you use Git's history and branching for data and models too, without bloating the repo, by keeping the heavy files in separate storage.

Learning Outcomes¶

  • Track a dataset with DVC.
  • Switch between data versions using git checkout + dvc checkout.
  • Set up a DVC remote and push/pull data.
  • Build and rerun a DVC pipeline.
  • Compare metrics between commits.

Question 8.1 (Solved): Track a dataset with DVC¶

In [ ]:
# Go to project
%cd /kaggle/working/movie-mood

# Configure Git
!git config --global user.name "Sarthak Patil"
!git config --global user.email "your-email@example.com"

# Create data directory
!mkdir -p data

# Copy dataset from Kaggle input
!cp /kaggle/input/datasets/sarthaksameerpatil/movies-labelled/movies_labelled.csv data/movies_labelled.csv

# Check dataset
!ls -lh data/

# DVC is ALREADY initialized, so don't run dvc init again

# Commit DVC initialization
!git add .dvc .dvcignore
!git commit -m "Initialise DVC"

# Track dataset with DVC
!dvc add data/movies_labelled.csv

# Commit DVC metadata
!git add data/movies_labelled.csv.dvc data/.gitignore
!git commit -m "Track dataset v1 with DVC"

# View DVC metadata
!cat data/movies_labelled.csv.dvc

Question 8.2: Make and switch between two data versions¶

  1. Track your dataset with DVC and commit (this is v1).
  2. Add 50 new labelled rows to the CSV.
  3. Run dvc add again, then git commit -m "Dataset v2".
  4. Now go back: git checkout <v1 commit> then dvc checkout.
  5. Print the number of rows in the CSV.

You should be back at the v1 row count. Paste your terminal output as proof.

In [ ]:
# YOUR ANSWER HERE
answer_8_2 = "..."

Question 8.3: Set up remote storage¶

DVC pushes the real files to a remote store. Use a local folder as a practice remote:

mkdir /tmp/dvc-store
dvc remote add -d myremote /tmp/dvc-store
git commit -am "Add DVC remote"
dvc push

Then answer: after dvc push, what is stored in GitHub and what is stored in the remote? Be specific.

In [ ]:
# YOUR ANSWER HERE
answer_8_3 = "..."

Question 8.4: Build a DVC pipeline¶

A pipeline records the steps that turn data into a model, so DVC knows what to rerun when something changes.

Create two stages:

dvc stage add -n prepare \
  -d src/prepare.py -d data/raw.csv \
  -o data/movies_labelled.csv \
  python src/prepare.py

dvc stage add -n train \
  -d src/train.py -d data/movies_labelled.csv \
  -o models/model.pt -M metrics.json \
  python src/train.py

dvc repro

Run it. Then paste the contents of the generated dvc.yaml.

In [ ]:
# YOUR CODE HERE

Question 8.5: Watch DVC skip work¶

Run dvc repro a second time without changing anything. What does it print, and why did it not retrain?

Now change one line in src/train.py and run dvc repro again. Which stages ran this time? Explain how DVC knew.

In [ ]:
# YOUR ANSWER HERE
answer_8_5 = "..."

Question 8.6: Compare metrics across versions¶

Use dvc metrics show and dvc metrics diff to compare metrics.json between two commits.

Paste the output and say in one line what it tells you that a plain git diff would not.

In [ ]:
# YOUR CODE HERE

10. Versioning Trained Models¶

A model file needs an identity too. final_model_v2_new.pt is not an identity.

A good model version records: which code commit, which data version, which settings, and which score.

Why is this important?¶

Without this record, you cannot answer "how was this model made" months later, and you cannot safely promote, compare, or roll back models.

Learning Outcomes¶

  • Save a model version with a model card that traces code, data, settings, and score.
  • Compare versions in a single table.
  • Promote a version to production and roll back.
  • Trace a production model back to a fully reproducible recipe.

Question 9.1 (Solved): A model card saved next to the model¶

In [ ]:
# SOLVED EXAMPLE
import subprocess
from datetime import datetime

def git_commit():
    try:
        out = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
                             capture_output=True, text=True)
        return out.stdout.strip() or "unknown"
    except Exception:
        return "unknown"

def save_model_version(model, version, metrics, hyperparams, data_path):
    folder = Path("models") / version
    folder.mkdir(parents=True, exist_ok=True)
    torch.save(model.state_dict(), folder / "model.pt")
    card = {
        "version": version,
        "created": datetime.now().isoformat(timespec="seconds"),
        "git_commit": git_commit(),
        "data_hash": file_hash(data_path),
        "hyperparams": hyperparams,
        "metrics": metrics,
        "stage": "staging",          # staging -> production -> archived
    }
    (folder / "model_card.json").write_text(json.dumps(card, indent=2))
    return card

# save_model_version(model, "v1.0", {"val_acc": 0.86},
#                    {"lr": 0.01, "hidden": 32},
#                    DATA_DIR / "movies_labelled.csv")

Question 9.2: Save three versions¶

Train and save three model versions:

  • v1.0 - hidden size 32,
  • v1.1 - hidden size 64,
  • v2.0 - hidden size 64, trained on the larger dataset from Q8.2.

Print all three model cards.

In [ ]:
# YOUR CODE HERE

Question 9.3: A naming rule¶

Look at your three versions. Write down the rule you used to decide when the number goes from 1.0 to 1.1, and when it goes to 2.0.

Then say which part of your rule handles a data change and which handles a code change.

In [ ]:
# YOUR ANSWER HERE
answer_9_3 = "..."

Question 9.4: Compare versions in one table¶

Write compare_versions() that reads every model_card.json under models/ and returns one DataFrame with: version, date, git commit, data hash, key hyperparameters, and validation accuracy.

Sort it by accuracy and print it.

In [ ]:
# YOUR CODE HERE
def compare_versions():
    pass

Question 9.5: Promote and roll back¶

Write two small functions:

  1. promote(version) - sets that version's stage to "production" and sets any other production model to "archived".
  2. current_production() - returns the version that is live right now.

Promote v1.1, then promote v2.0, then roll back to v1.1. Print the stage of all three versions after each step.

In [ ]:
# YOUR CODE HERE

Question 9.6: The full trace¶

Pick your production model. Starting from only its model card, write down the exact steps a new teammate would follow to rebuild that model from scratch.

If any step is impossible with what you saved, that is a gap - name it and add the missing field to your model card.

In [ ]:
# YOUR ANSWER HERE
answer_9_6 = "..."

11. Putting It Together¶

Why is this important?¶

Real projects need every piece from this lab working together at once - config-driven, checkpointed, tracked, versioned, and resumable. This section proves you can combine them.

Learning Outcomes¶

  • Combine checkpointing, tracking, early stopping, and model versioning into a single script.
  • Prove a training run can survive an interruption and resume cleanly.

Question 10.1: One tracked, versioned, resumable run¶

Combine everything into a single script train_v2.py that:

  1. Reads settings from config.json.
  2. Sets all seeds (Week 3).
  3. Prints the data hash and the git commit.
  4. Trains with checkpointing every 10 epochs, saving best.pt and last.pt.
  5. Uses early stopping.
  6. Logs every epoch to MLflow (or W&B).
  7. Saves a model card at the end.
  8. Can be restarted with --resume and continues from the last checkpoint.
In [ ]:
# YOUR CODE HERE

Question 10.2: The interruption test¶

Run train_v2.py. Press Ctrl+C at around epoch 40. Then run it again with --resume.

Show the log lines proving it continued from epoch 40 and not from 0. Confirm that the MLflow curve is continuous, with no reset.

In [ ]:
# YOUR ANSWER HERE
answer_10_2 = "..."

11. Challenge Problems (Optional)¶

These go beyond the core lab. Attempt any of them for extra practice.

Challenge 11.1: Checkpoint size Save a checkpoint with and without the optimiser state and compare file sizes. Then try torch.save with half precision weights. How much smaller is it, and what is the risk?

Challenge 11.2: Automatic experiment tags Write a Git pre-commit hook (or a small script) that refuses to commit if metrics.json was changed but model_card.json was not. This forces the two to stay in step.

Challenge 11.3: DVC with real cloud storage Set up a DVC remote on Google Drive or an S3-compatible store instead of a local folder. Push your data, delete the local copy, and pull it back on a fresh clone.

Challenge 11.4: Reproduce a classmate's number Take a classmate's GitHub repo and DVC remote. Clone, dvc pull, and rerun. Do you get their exact accuracy? Write down every difference you had to fix.

In [ ]:
# Optional: attempt any of the challenge problems above here
# YOUR CODE / ANSWER HERE