Bin Analogy ML

Imports¶

Part 1: Bin Analogy for Machine Learning¶

Cell 1.1: Visual bin: population of marbles¶

Goal:

• Visualize an unknown population of marbles in a bin, and introduce $\mu$ as the population’s true, unknown proportion of red marbles

Implementation: cell1_draw_bin_with_marbles_interactive()

• Places a grid of marbles, colors a mu fraction of them red with _draw_bin_with_marbles(), and shuffles the colors with seed so the arrangement looks random while the proportion stays fixed

Usage

• Inputs

  • seed: random seed for the spatial arrangement of marbles
  • mu: true proportion of red marbles in the population, 0-1

• Panels

  • Bin with marbles: the full population, red and green marbles laid out on a grid, with mu and the red count in the title

Guided usage

• Change seed several times, leaving mu fixed
  • Observe the marbles reshuffle into a new arrangement, but the total red count stays exactly $\mu \times$ total marbles every time
• Raise mu from 0 toward 1
  • Observe the bin fill up with more red marbles, since $\mu$ is the fixed population proportion, not something observed by sampling
• In real-world scenarios, $\mu$ is never known directly, only estimated from samples: the rest of this notebook builds that estimation problem

Cell 1.2: Single experiment: is $\nu$ close to $\mu$?¶

Goal:

• Draw one sample of N marbles from the bin, compute the sample proportion $\nu$, and see how close a single estimate lands to $\mu$

Implementation: cell2_plot_single_experiment_interactive()

• Draws N Bernoulli(mu) samples with _plot_single_experiment(), and computes $\nu = \frac{1}{N}\sum_{i=1}^{N} x_i$
• Colors the comparison bar chart green, yellow, or red by how far $\nu$ landed from $\mu$

Usage

• Inputs

  • seed: random seed for the sample drawn
  • mu: true proportion of red marbles in the population, 0-1
  • N: number of marbles sampled, 10-1000

• Panels

  • Population vs sample: bar chart comparing $\mu$ against $\nu$, colored by how close the two are
  • Interpretation: current parameters, the sampled $\nu$, and the error $|\nu - \mu|$

Guided usage

• Change seed several times, leaving mu and N fixed
  • Observe $\nu$ land in a different place each time, and the bar color flip between green, yellow, and red: one experiment is not reliable on its own
• Raise N from 10 toward 1000, leaving seed fixed
  • Observe $\nu$ settle closer to $\mu$ and the bar turn green more often
• A single experiment only gives a point estimate, never a sense of how reliable that estimate is
  • What is really needed is $P(|\nu - \mu| > \epsilon)$: the probability that an estimate this far off would happen at all
  • Repeating the experiment many times, next, is how that probability gets measured

Cell 1.3: Monte Carlo simulation: distribution of $\nu$¶

Goal:

• Repeat the sampling experiment many times to see the full distribution of $\nu$, and empirically estimate $P(|\nu - \mu| > \epsilon)$

Implementation: cell3_monte_carlo_simulation_interactive()

• Runs n_experiments independent trials of N Bernoulli(mu) samples each in _plot_monte_carlo_simulation(), recording $\nu$ per trial
• Overlays a KDE on the $\nu$ histogram, shades the region where $|\nu - \mu| > \epsilon$, and reports the fraction of trials landing there

Usage

• Inputs

  • seed: random seed for the experiments
  • mu: true proportion of red marbles in the population, 0-1
  • N: samples per experiment, 10-500
  • n_experiments: number of repeated experiments, 100-10000
  • eps: tolerance $\epsilon$ defining “far from $\mu$”

• Panels

  • Distribution of nu: histogram plus KDE of $\nu$ across experiments, true mu marked, and the $|\nu - \mu| > \epsilon$ tails shaded red
  • Key insights: current parameters, $\text{mean}(\nu)$, $\text{std}(\nu)$, and $P(|\nu - \mu| > \epsilon)$

Guided usage

• Raise N from 10 toward 500, leaving n_experiments fixed
  • Observe the histogram narrow around mu and $P(|\nu - \mu| > \epsilon)$ drop: this is the Law of Large Numbers, $\nu \xrightarrow{P} \mu$ as $N \to \infty$
• Raise n_experiments toward 10000
  • Observe the histogram fill in and its shape approach the smooth normal curve predicted by the Central Limit Theorem
• Replace “marbles in a bin” with “data points”, $\mu$ with generalization error, and $\nu$ with training error on N samples
  • The same result then reads: with enough training samples, training error is close to generalization error, the connection this bin analogy sets up for the rest of the course