Overfitting - MSML610 -- Tutorials

Imports¶

%load_ext autoreload
%autoreload 2

import logging

import matplotlib.pyplot as plt
import seaborn as sns

# Set plotting style.
sns.set_style("whitegrid")
plt.rcParams["figure.figsize"] = (12, 6)

import helpers.hintrospection as hintros
import helpers.htutorial as ut
import L05_02_02_overfitting_utils as utils

ut.config_notebook()

# Initialize logger.
logging.basicConfig(level=logging.INFO)
_LOG = logging.getLogger(__name__)

WARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
vim support installed: restart the notebook, if needed

Python 3.12.3
Linux 589569fe8102 6.12.67-linuxkit #1 SMP Sun Jan 25 02:26:28 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux

Part 1: Overfitting: Data, Models, and Generalization¶

Cell 1.1: True target function and data sampling¶

Goal:

Visualize an unknown target function $f(x)$ , sample noisy observations from it, and split them into training and test data, the basic setup every learning problem in this notebook starts from

Implementation: cell1_plot_true_target_function()

Samples N points from the chosen Function, adds Gaussian noise of scale epsilon, and splits them 80/20 into training and test sets
Stores the split in shared state so Cell 1.2 fits models against the same data

hintros.print_obj_info(utils.cell1_plot_true_target_function)

Usage

Inputs
- seed: random seed for the sampled points and the train/test split
- Function: target function $f$ : slow sinusoid, fast sinusoid, parabola, constant, or linear
- epsilon: standard deviation of the observation noise, 0-1
- N (total samples) (log scale): total points sampled, 4-1024
Panels
- True target function: the noiseless $f(x)$ , plus a noisy overlay when epsilon > 0
- In-sample data (80%): the training points
- Out-of-sample data (20%): the test points
- Comments: current parameters and the train/test split sizes

# Display the true target function with interactive controls.
utils.cell1_plot_true_target_function()

Guided usage

Switch Function from Slow Sinusoid to Fast Sinusoid, leaving everything else fixed
- Observe the same N points now trace a much busier curve, since the function itself oscillates faster
Raise epsilon from 0 toward 1
- Observe the training and test points scatter further from the true curve, foreshadowing why $E_{in}$ alone will not tell the full story in Cell 1.2

Cell 1.2: Model comparison: constant vs linear¶

Goal:

Fit a constant or linear model to Cell 1.1’s training split, and compare in-sample error $E_{in}$ against out-of-sample error $E_{out}$

Implementation: cell2_plot_model()

Fits $h(x) = b$ (hypothesis class $\mathcal{H}_0$ ) or $h(x) = ax + b$ (hypothesis class $\mathcal{H}_1$ ) to the training data from Cell 1.1’s shared state, depending on Model Type
Resample and Relearn draws a fresh training/test split from the same Function/epsilon/N and refits

hintros.print_obj_info(utils.cell2_plot_model)

Usage

Inputs
- Model Type: Constant ( $h(x) = b$ ) or Linear ( $h(x) = ax + b$ )
- Resample and Relearn: draws a new training/test split and refits
Panels
- In-sample data and model: training points, the fitted $h(x)$ , and $E_{in}$
- Out-of-sample data and model: test points, the same $h(x)$ , and $E_{out}$
- True function vs model: $f(x)$ against $h(x)$ , with the approximation error shaded
- Comments: learned parameters, $E_{in}$ , and $E_{out}$

# Display model learning with interactive controls.
utils.cell2_plot_model()

Guided usage

Click Resample and Relearn several times on Constant
- Observe $h(x)$ barely moves between draws: low variance, but the shaded error against $f$ stays large: high bias
Switch to Linear and repeat
- Observe $h(x)$ shift more between draws (higher variance), while the shaded error against $f$ shrinks (lower bias): the same bias-variance tradeoff as before, now seen on real train/test splits
Run Cell 1.1 again with a new Function or seed before revisiting this cell: it always fits against whatever split Cell 1.1 last produced