1. Ax - Multi-Objective Optimization for Marketing Campaigns

1. Introduction to Bayesian Optimization¶

1.1. Optimization Problem¶

We have an optimization problem:

$\min_{x} f(x)$

Where $f(x)$ is the objective function to minimize.

Gradient method -> $x_{t+1} = x_t - \alpha \nabla f(x_t)$

Requires a differentiable objective function!

1.2. Black-Box Optimization¶

In a black-box optimization problem, we don’t have a differentiable objective function. The relation between input and output is unknown.

This relation is modeled as an unknown function $f(x)$ that predicts the outcome.

Surrogate model: $y = f(x) + \epsilon$

How do we obtain the surrogate model?

1.2.1. Surrogate Model¶

Given hyperparameters $x$ , we want obtain a function $f(x)$ that predicts the outcome of the objective function.

We have a set of points $x_1, x_2, ..., x_n$ and the corresponding outcomes $y_1, y_2, ..., y_n$ .

1.2.2. Gaussian Process (GP)¶

Define $f(x) \sim GP(m(x), k(x, x'))$ where:

$m(x)$ is the mean function
$k(x, x')$ is the covariance function

Most common kernel: RBF kernel $k(x, x') = \sigma^2 \exp(-\frac{||x - x'||^2}{2l^2})$

Gaussian kernel
Smoothness

In other words:

Mean defined at each point $x_i$
Uncertainty defined at each point $x_i$
Covariance allows interpolation of unknown $f(x_i)$ , with some uncertainty

1.3. Bayesian Optimization¶

Bayesian optimization is a sequential optimization technique that uses a probabilistic model to guide the search for the optimal solution. In order to do this, first, an Acquisition Function has to be defined.

1.3.1. Acquisition Function¶

The acquisition function is the function used to define the next point to evaluate.

Note: $f^* = \max_i y_i$ -> Best observed outcome

Different types of acquisition functions are:

Expected Improvement (EI): $EI(x)=E[\max(f(x)−f^*, 0)]$

Probability of Improvement (PI): $PI(x)=P(f(x) \geq f^*)$

Upper Confidence Bound (UCB): $UCB(x) = \mu(x) + \kappa \sigma(x)$

Expected Improvement is the acquisition function used by Ax.

1.3.2. Sequential Optimization¶

Bayesian optimization leads to a sequential optimization process where the Acquisition Function is applied to define the next point to evaluate. After the evaluation, the result is fed back to the model to adjust the Surrogate Model.

2. Real-Time Bidding Algorithms with Bayesian Optimization¶

2.1. Real-Time Bidding (RTB)¶

Real-Time Bidding (RTB) is a programmatic advertising technique where advertisers bid on ad inventory in real-time. The advertiser pays only when the ad is clicked.

There are four actors in the RTB process:

Publisher: The website or app that sells the ad inventory.
Advertiser: The advertiser that wants to display their ad.
Ad Exchange: Works on the Publisher side. It’s the platform that starts the auction when a space is available, receives the bids from the DSP and decides which ad will be published.
DSP (Demand Side Platform): Works on the Advertiser side. It receives the auction request from the Ad Exchange and makes a bid to show the ad.

The Ad Exchange shares the Auction with multiple DSPs. Each DSP, which represents an advertiser, makes a bid. The highest bid wins the auction and the ad is published.

Note: The bid done by the DSP is not the final price of the Impression. A common practice is the use the second highest bid as the price of the Impression. This is called the “Second Price Auction”.

A different but more simple diagram:

2.2. Acronyms¶

DSP: Demand Side Platform. Works on the Advertiser side. It’s what we’re going to optimize.
RTB: Real-Time Bidding. The process of bidding for an impression in real-time.
CTR: Click-Through Rate. The probability of a user clicking on an ad.
CVR: Conversion Rate. The probability of a user converting (Buying) given that they clicked on the ad.
CPC: Cost Per Click. The cost of a click.
CPM: Cost Per Mile Impressions. The cost of 1000 impressions.
eCPC: effective Cost Per Click (The value paid for the impression is usually not the bid price, but the second highest bid)
pCTR: predicted Click-Through Rate. The probability of a user clicking on an ad, predicted by a machine learning model.

2.3. Bidding Strategy¶

A DSP is running a campaign for an advertiser. It receives an auction and has to make a bid. A bidding strategy is used to determine the bid amount. The DSP wants to maximize the revenue while minimizing the cost for the advertiser. The revenue can be measured in Impressions, Clicks, or Conversions, depending on the campaign objective.

The steps in the bidding strategy are:

Learn from existing data, this includes information about the publisher, users, cookies, the content to advertise, etc.
Receive an auction request from the Ad Exchange.
Predict a potential CTR
Define a formula to calculate the bid amount. If it’s to high, we may be spending the advertiser’s budget without getting good results. If it’s to low, we may be losing the auction.

Important: Given that we had predicted the CTR, we want to calculate the bid amount to an optimal value. This is what we’re trying to achieve.

Classic algorithms to define the bid amount are:

Constant bidding (Const): Bid a constant value for all the bid requests
Random bidding (Rand): Randomly choose a bid value in a given range.
Bidding below max eCPC (Mcpc): The goal of bid optimisation is to reduce the eCPC (Effective Cost Per Click). The Mcpc is what the advertiser is willing to pay for the impression at most. The bid price on an impression is obtained by multiplying the Mcpc and the pCTR.
Linear-form bidding of pCTR (Lin): Bid a linear function of the predicted CTR. The bid amount increases with a higher predicted CTR.

Lin has demonstrated to be one of the best and simplest bidding strategies.

2.3.1. Linear-form bidding of pCTR (Lin)¶

The basic formula is:

$bid = base\_bid + \frac{pCTR}{avg\_CTR}$

$base\_bid$ : A constant value for all the bid requests. The average bid value of the campaign.
$pCTR$ : The predicted Click-Through Rate.
$avg\_CTR$ : The average Click-Through Rate of the campaign, or the advertiser.

The rationale within this formula is that a campaign naturally has an average probability of a click (Average CTR). Some bids, based on the data provided by the publisher, may have a higher probability of a click. On those we want to bid higher.

The coefficient $ctr\_reg\_coef$ is used to regulate how much the predicted CTR is going to affect the final bid price. This is a hyperparameter that can be tuned to optimize the campaign. It’s between 0.5 and 2.

In order to improve this bidding algorithm, we’ll introduce more concepts.

2.3.1.1. Base Bid¶

$base\_bid$ represents how much the DSP (Or the advertiser) is willing to pay per thousand impressions of average quality.

2.3.1.2. Pay to Bid Ratio¶

As explained in the previous section, the bid price is not the final price of the impression. The final price is the second highest bid. This means we can always aim a little bit higher to win the auction as it’s not the price we’re going to pay.

To take advantage of this, we introduce the concept of “Pay to Bid Ratio”. Which means, how much more we can bid in order to win the auction assuming we’re not going to pay the full price.

Complex machine learning models can be used to predict how much other DSPs could be bidding for the same impression. In this case we’ll just keep a simple ratio.

$pay\_to\_bid\_ratio = \frac{raw\_bid}{pay\_bid}$

If the pay price is much lower than the raw bid, the ratio will be higher. This ratio is always greater than 1. This means we can increase the bid price to win the auction.

In this case we’ll keep an average of the historic ratio.

$pay\_to\_bid\_ratio$ is clipped between 0.5 and 2.

The coefficient $pay\_to\_bid\_reg\_coef$ is used to regulate how much the pay to bid ratio is going to affect the final bid price. This is a hyperparameter that can be tuned to optimize the campaign. It’s also limited between 0.5 and 2.

$final\_bid = raw\_bid \times (1 + pay\_to\_bid\_reg\_coef \times (pay\_to\_bid\_ratio - 1))$

$raw\_bid$ is the bid amount obtained from the basic “Lin” formula.

2.3.1.3. Pacing¶

The advertiser usually sets a budget for the campaign. The DSP needs to ensure that the bid amount is within the budget and calculate how much is left. This introduces the concept of “pay_price”, which is the price paid for an impression, not the price of the bid.

If the base_bid is too high, the budget could be consumed early in the day, while the majority of the conversions happen at night. To correct this, the DSP can use “pacing”. The pacing is a coefficient calculated over the time of the day based on the expected budget to be spent.

$pacing\_coefficient$ : A coefficient between 0.5 and 2 that is calculated over the time of the day based on the expected budget to be spent.

$pacing\_coefficient = \frac{expected\_budget\_to\_be\_spent}{actual\_budget\_spent}$

With $expected\_budget\_to\_be\_spent = \frac{budget}{24} \times hour\_of\_day$

This means, we try to fraction the budget for each hour of the day.

$pacing\_coefficient$ is clipped between 0.5 and 2.

$final\_bid = raw\_bid \times pacing\_coefficient$

The coefficient $pacing\_reg\_coef$ is used to regulate how much the pacing is going to affect the final bid price. This is a hyperparameter that can be tuned to optimize the campaign. It’s also limited between 0.5 and 2.

2.3.1.4. Final Bid Formula¶

The final bid formula without the regulation coefficients is:

$final\_bid = base\_bid \times \frac{pCTR}{avg\_CTR} \times pay\_to\_bid\_ratio \times pacing\_coefficient$

With the regulation coefficients:

$final\_bid = base\_bid \times \left(1 + ctr\_reg\_coef \times \left(\frac{pCTR}{avg\_CTR} - 1\right)\right) \times \left(1 + pay\_to\_bid\_reg\_coef \times (pay\_to\_bid\_ratio - 1)\right) \times \left(1 + pacing\_reg\_coef \times (pacing\_coefficient - 1)\right)$

Hyperparameters:
- $base\_bid$ : How much the DSP (Or the advertiser) is willing to pay per thousand impressions of average quality
- $ctr\_reg\_coef$ : A coefficient to regulate the importance of the predicted CTR. (0.5 to 2)
- $pay\_to\_bid\_reg\_coef$ : A coefficient to regulate the importance of the pay to bid ratio. (0.5 to 2)
- $pacing\_reg\_coef$ : A coefficient to regulate the importance of the pacing. (0.5 to 2)
- $budget$ : The budget set by the advertiser.
Calculated values:
- $pCTR$ : The predicted Click-Through Rate.
- $avg\_CTR$ : The average Click-Through Rate of the campaign, or the advertiser.
- $pay\_to\_bid\_ratio$ : The ratio of the pay price to the raw bid.
- $pacing\_coefficient$ : The coefficient to regulate the pacing.

2.4. DSP Simulation¶

2.4.1. iPinYou Dataset¶

This simulation is based on the iPinYou dataset, described in these publications:

iPinYou Global RTB Bidding Algorithm Competition Dataset (Hairen Liao, Lingxiao Peng, Zhenchuan Liu, Xuehua Shen)
Real-Time Bidding Benchmarking with iPinYou Dataset (Weinan Zhang, Shuai Yuan, Jun Wang)

For more information:

iPinYou is a DSP (Demand Side Platform) for RTB (Real-Time Bidding) in China that published in 2014 a complete dataset of individual bids, impressions and clicks for a certain group of advertisers during a week.

iPinYou opened this dataset for a competition to create the best bidding algorithm. A bidding algorithm is a function that, based on some information given by the Ad Exchange, decides whether to bid for an impression and the bid amount.

The analysis and processing of the dataset is done in the Jupyter Notebook dsp_pctr_prediction_model.ipynb.

Important: The bid and paid price are measured in CNY (Chinese Yuan) per 1000 impressions.

2.4.2. Prediction Model¶

A requirement for the simulation is to predict the Click-Through Rate (CTR) of an impression. This is a classic machine learning problem. A model using XGBoost has been created to predict the CTR based on the auction request provided by the Ad Exchange.

The model training and evaluation is done in the Jupyter Notebook dsp_pctr_prediction_model.ipynb.

The dataset has been updated to include the predicted CTR. This new feature will be used by the DSP to calculate the bid amount. The dataset is available in the dataset folder.

2.4.3. Dataset¶

The iPinYou dataset has been reduced to a smaller dataset files (From 4GB to 300MB) to only contain the features that are used to predict the CTR, plus other features required for the simulation:

Paid price
Clicked or not
Floor price

Original iPinYou dataset contains:

Bid logs: Every auction request from the Ad Exchange, with some data about the ad, the amount bidded by the DSP.
Impression logs: Every won bid generates an impression, this adds the paid price data.
Click logs: Some impressions generate a click, this is logged here.

The new dataset is reduced to a file per day. The dataset contains all this information for one advertiser and from June 6th 2013 to June 12th 2013.

The dataset is available in the dataset folder.

2.4.4 Simulation for 06/10/2013 with fixed hyperparameters¶

The simulation will run for a given date (E.g. 06/10/2013). It will start processing the dataset which contains one line per bid. Then, for each bid:

Predict the CTR using the prediction model.
Calculate the bid amount using the bidding strategy.
Verify if the bid was won, based on the paid price and the floor price.

The base_bid, budget and coefficients are fixed for the simulation.

Initialize environment

%reload_ext autoreload
%load_ext autoreload
%autoreload 2
%matplotlib inline

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

# Install AX with the extension for Jupyter notebooks
! pip3 install "ax-platform[notebook]" joblib --break-system-packages

Fetching long content....

import numpy as np
from ax.api.client import Client
from ax.api.configs import RangeParameterConfig
import matplotlib.pyplot as plt
import pandas as pd
from IPython.display import display
from Ax_utils import (
    AB_Testing,
    UCB1,
    ThompsonSampling,
    GP_Bandit,
    SimulationExperiment,
    dsp_simulation,
    get_avg_spent_per_day,
)

Download dataset

If the files are not available in the dataset folder, download them

# Download dataset
! wget -P dataset/ https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130607.csv
! wget -P dataset/ https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130608.csv
! wget -P dataset/ https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130609.csv
! wget -P dataset/ https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130610.csv
! wget -P dataset/ https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130611.csv
! wget -P dataset/ https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130612.csv

--2026-03-02 09:10:59--  https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130607.csv
Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.16.207, 172.253.63.207, 142.251.163.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.16.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 56479182 (54M) [text/csv]
Saving to: ‘dataset/bid_with_features_and_pctr_20130607.csv.1’

bid_with_features_a 100%[===================>]  53.86M  54.5MB/s    in 1.0s    

2026-03-02 09:11:00 (54.5 MB/s) - ‘dataset/bid_with_features_and_pctr_20130607.csv.1’ saved [56479182/56479182]

--2026-03-02 09:11:00--  https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130608.csv
Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.31.207, 172.253.63.207, 142.251.163.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.31.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48926639 (47M) [text/csv]
Saving to: ‘dataset/bid_with_features_and_pctr_20130608.csv.1’

bid_with_features_a 100%[===================>]  46.66M  54.5MB/s    in 0.9s    

2026-03-02 09:11:01 (54.5 MB/s) - ‘dataset/bid_with_features_and_pctr_20130608.csv.1’ saved [48926639/48926639]

--2026-03-02 09:11:01--  https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130609.csv
Resolving storage.googleapis.com (storage.googleapis.com)... 172.253.115.207, 172.253.63.207, 172.253.122.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.115.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50058820 (48M) [text/csv]
Saving to: ‘dataset/bid_with_features_and_pctr_20130609.csv.1’

bid_with_features_a 100%[===================>]  47.74M  51.9MB/s    in 0.9s    

2026-03-02 09:11:03 (51.9 MB/s) - ‘dataset/bid_with_features_and_pctr_20130609.csv.1’ saved [50058820/50058820]

--2026-03-02 09:11:03--  https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130610.csv
Resolving storage.googleapis.com (storage.googleapis.com)... 192.178.155.207, 172.253.63.207, 172.253.122.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|192.178.155.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 51285781 (49M) [text/csv]
Saving to: ‘dataset/bid_with_features_and_pctr_20130610.csv.1’

bid_with_features_a 100%[===================>]  48.91M  57.2MB/s    in 0.9s    

2026-03-02 09:11:04 (57.2 MB/s) - ‘dataset/bid_with_features_and_pctr_20130610.csv.1’ saved [51285781/51285781]

--2026-03-02 09:11:04--  https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130611.csv
Resolving storage.googleapis.com (storage.googleapis.com)... 172.253.139.207, 172.253.63.207, 172.253.122.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.139.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 51538095 (49M) [text/csv]
Saving to: ‘dataset/bid_with_features_and_pctr_20130611.csv.1’

bid_with_features_a 100%[===================>]  49.15M  54.2MB/s    in 0.9s    

2026-03-02 09:11:05 (54.2 MB/s) - ‘dataset/bid_with_features_and_pctr_20130611.csv.1’ saved [51538095/51538095]

--2026-03-02 09:11:06--  https://storage.googleapis.com/msml610-assets/bid_with_features_and_pctr_20130612.csv
Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.111.207, 172.253.63.207, 172.253.122.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.111.207|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 52883893 (50M) [text/csv]
Saving to: ‘dataset/bid_with_features_and_pctr_20130612.csv.1’

bid_with_features_a 100%[===================>]  50.43M  53.4MB/s    in 0.9s    

2026-03-02 09:11:07 (53.4 MB/s) - ‘dataset/bid_with_features_and_pctr_20130612.csv.1’ saved [52883893/52883893]

Restrict the budget

The machine learning model created in “dsp_pctr_prediction_model.ipynb” also extracts some information about the dataset. Part of this information is how much has been spent by the DSP in the real scenario.

The number of bids we can simulate is limited by the number of bids in the dataset. For this reason, we’ll restrict the daily budget to a percentage of budget spent in the real scenario.

If we don’t restrict the budget, we could run into an escenario where ads budget is left but we don’t have more information to simulate the bidding.

dsp_simulation

The dsp_simulation is available in the Ax_utils.py file.

The function receives the date, and the hyperparameters. Then, as it goes through the bid requests it predicts the CTR (Using the created XGBoost model), calculates the bid amount, considering the pay to bid ratio, and pacing. Then, it compares the calculated bid with the price that was really paid in the dataset. If it’s a win, it considers the dataset paid price for the budget, not the bid, as the second highest bid is the one used for the payment.

Simulation #1

Base bid: 50
Date: 2013/06/10

avg_spent_per_day = get_avg_spent_per_day("20130610")
# The budget has to be much lower than the budget spent per day, otherwise we'll run out of bids in the simulation without exhausting the budget and we cannot keep trying.
budget = avg_spent_per_day * 0.2

total_bids, total_bids_done, total_bids_won, total_budget_spent, total_clicks = (
    dsp_simulation("20130610", 50, budget, 0.0, 0.0, 0.0)
)

print(f"Total budget: {budget}")
print(f"Total bids: {total_bids}")
print(f"Total bids done: {total_bids_done}")
print(f"Total bids won: {total_bids_won}")
print(f"Total budget spent: {total_budget_spent}")
print(f"Total clicks: {total_clicks}")

Total budget: 6019776.0
Total bids: 434258
Total bids done: 313310
Total bids won: 179127
Total budget spent: 5615432.0
Total clicks: 79

Now I’ll run a simulation where I pay more per click. The budget will be consumed faster, but we have better chances of winning bids.

Simulation #2

Base bid: 100
Date: 2013/06/10

total_bids, total_bids_done, total_bids_won, total_budget_spent, total_clicks = (
    dsp_simulation("20130610", 100, budget, 0.0, 0.0, 0.0)
)
print(f"Total budget: {budget}")
print(f"Total bids: {total_bids}")
print(f"Total bids done: {total_bids_done}")
print(f"Total bids won: {total_bids_won}")
print(f"Total budget spent: {total_budget_spent}")
print(f"Total clicks: {total_clicks}")

Total budget: 6019776.0
Total bids: 142765
Total bids done: 137453
Total bids won: 117052
Total budget spent: 6019779.0
Total clicks: 83

The number of total bids won is lower because the budget was consumed faster.

Keep increasing the amount to pay per click.

Simulation #3

Base bid: 400
Date: 2013/06/10

total_bids, total_bids_done, total_bids_won, total_budget_spent, total_clicks = (
    dsp_simulation("20130610", 400, budget, 0.0, 0.0, 0.0)
)
print(f"Total budget: {budget}")
print(f"Total bids: {total_bids}")
print(f"Total bids done: {total_bids_done}")
print(f"Total bids won: {total_bids_won}")
print(f"Total budget spent: {total_budget_spent}")
print(f"Total clicks: {total_clicks}")

Total budget: 6019776.0
Total bids: 86455
Total bids done: 86455
Total bids won: 86455
Total budget spent: 6019782.0
Total clicks: 76

The number of bids won and clicks went down. We consumed the budget much faster.

Let’s try some enabling the CTR coefficient, which should make this more efficient (It makes use of the predicted CTR)

Simulation #4

Base bid: 400
Date: 2013/06/10
CTR coefficient: 1.0

total_bids, total_bids_done, total_bids_won, total_budget_spent, total_clicks = (
    dsp_simulation("20130610", 400, budget, 1.0, 0.0, 0.0)
)
print(f"Total budget: {budget}")
print(f"Total bids: {total_bids}")
print(f"Total bids done: {total_bids_done}")
print(f"Total bids won: {total_bids_won}")
print(f"Total budget spent: {total_budget_spent}")
print(f"Total clicks: {total_clicks}")

Total budget: 6019776.0
Total bids: 93004
Total bids done: 92935
Total bids won: 90611
Total budget spent: 6019851.0
Total clicks: 78

Great, the number of bids won and clicks went up. It means the CTR prediction is helping.

I’ll try to increase bid to pay ratio coefficient, this may help to win some bids even when the paid price won’t increase.

Simulation #5

Base bid: 400
Date: 2013/06/10
CTR coefficient: 1.0
Bid to pay ratio coefficient: 1.0

total_bids, total_bids_done, total_bids_won, total_budget_spent, total_clicks = (
    dsp_simulation("20130610", 400, budget, 1.0, 1.0, 0.0)
)
print(f"Total budget: {budget}")
print(f"Total bids: {total_bids}")
print(f"Total bids done: {total_bids_done}")
print(f"Total bids won: {total_bids_won}")
print(f"Total budget spent: {total_budget_spent}")
print(f"Total clicks: {total_clicks}")

Total budget: 6019776.0
Total bids: 86476
Total bids done: 86475
Total bids won: 86471
Total budget spent: 6019806.0
Total clicks: 76

It didn’t help. The budget was consumed faster.

Now, what happens if I increase the bid to pay ratio coefficient when the bid base was much lower (50 vs 400)

Simulation #6

Base bid: 50
Date: 2013/06/10
CTR coefficient: 1.0
Bid to pay ratio coefficient: 1.0

total_bids, total_bids_done, total_bids_won, total_budget_spent, total_clicks = (
    dsp_simulation("20130610", 50, budget, 1.0, 1.0, 0.0)
)
print(f"Total budget: {budget}")
print(f"Total bids: {total_bids}")
print(f"Total bids done: {total_bids_done}")
print(f"Total bids won: {total_bids_won}")
print(f"Total budget spent: {total_budget_spent}")
print(f"Total clicks: {total_clicks}")

Total budget: 6019776.0
Total bids: 107065
Total bids done: 106156
Total bids won: 97609
Total budget spent: 6019791.0
Total clicks: 81

Here, with a lower bid base, the bid to pay ratio helped. We can see the relationship is not linear, and they aren’t independent.

Now we’ll try increasing the pacing coefficient, without the pay to bid ratio.

Simulation #7

Base bid: 50
Date: 2013/06/10
CTR coefficient: 1.0
Pacing coefficient: 1.0

total_bids, total_bids_done, total_bids_won, total_budget_spent, total_clicks = (
    dsp_simulation("20130610", 50, budget, 1.0, 0.0, 1.0)
)
print(f"Total budget: {budget}")
print(f"Total bids: {total_bids}")
print(f"Total bids done: {total_bids_done}")
print(f"Total bids won: {total_bids_won}")
print(f"Total budget spent: {total_budget_spent}")
print(f"Total clicks: {total_clicks}")

Total budget: 6019776.0
Total bids: 255178
Total bids done: 204364
Total bids won: 128395
Total budget spent: 6019820.0
Total clicks: 103

Here the pacing was incredibly helpful. We got the best results among all the simulations. It was totally unexpected.

2.4.4.1 Analysis¶

The results for different hyperparameters are very different. Each parameter affects how the other parameters perform. This couldn’t be analyzed with a simple linear regression.

Now, in a real experiment we can’t do this, we cannot go back in time to see how different parameters perform. We need to do adaptive experiments. For Adaptive Experimentation, we can use Bayesian Optimization.

2.4.5. Bayesian Optimization¶

Experiment Simulation

In this section we’re not going to simulate using the same dataset again and again. We’re going to simulate the experiment as if it was real. We cannot go back in time to see how different parameters perform.

The simulation will be run for a given date (E.g. 06/07/2013). Then, it will run for a new set of hyperparameters the next day, and so on.

Note: We only have data for 6 days of testing. A Trial is set to last 1 day. Running a simulation period of less than 1 day won’t work properly as we want to maximize the number of clicks, and the number of clicks in one day is already too low. To test a better simulation we do a rolling use of the dataset, starting again from the first day.

The simulation will run for a given date (E.g. 06/10/2013). It will start processing the dataset which contains one line per bid request. Then, for each bid request:

Predict the CTR using the prediction model.
Calculate the bid amount using the bidding strategy.
Verify if the bid was won, based on the paid price and the floor price.
Update the budget based on the paid price.
If it was clicked, update the CTR and average CTR.

About the model predictions:

The pCTR can only be predicted with the information from the previous days.
If the current simulation day is 06/10/2013, the model 2013-06-06_2013-06-09 will be loaded

2.4.5.1 Maximize the number of clicks¶

In this case the budget is fixed to 6019776.0 and the goal is to maximize the number of clicks.

budget = 6019776.0

client = Client()

parameters = [
    RangeParameterConfig(
        name="base_bid", parameter_type="float", bounds=(20, 100)
    ),
    RangeParameterConfig(
        name="bid_to_pay_ratio", parameter_type="float", bounds=(0, 1)
    ),
    RangeParameterConfig(
        name="ctr_coefficient", parameter_type="float", bounds=(0, 1)
    ),
    RangeParameterConfig(
        name="pacing_coefficient", parameter_type="float", bounds=(0, 1)
    ),
]

client.configure_experiment(parameters=parameters)
client.configure_optimization(objective="clicks")

# The experiment runs over 18 days. The dataset only contains 6 days of data, so we roll the dataset.
for date in [
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
]:
    trials = client.get_next_trials(max_trials=1)
    trial_index = list(trials.keys())[0]
    parameters = trials[trial_index]

    # These are the optimized parameters
    base_bid = parameters["base_bid"]
    bid_to_pay_ratio = parameters["bid_to_pay_ratio"]
    ctr_coefficient = parameters["ctr_coefficient"]
    pacing_coefficient = parameters["pacing_coefficient"]

    print(
        f"Simulating date: {date} with base_bid: {base_bid}, bid_to_pay_ratio: {bid_to_pay_ratio}, ctr_coefficient: {ctr_coefficient}, pacing_coefficient: {pacing_coefficient}"
    )
    (
        total_bids,
        total_bids_done,
        total_bids_won,
        total_budget_spent,
        total_clicks,
    ) = dsp_simulation(
        date,
        base_bid,
        budget,
        bid_to_pay_ratio,
        ctr_coefficient,
        pacing_coefficient,
    )
    print(
        f"Total clicks: {total_clicks}, impressions: {total_bids_won}, budget spent: {total_budget_spent}"
    )

    client.complete_trial(
        trial_index=trial_index,
        raw_data={
            "clicks": total_clicks,
            "impressions": total_bids_won,
            "budget_spent": total_budget_spent,
        },
    )

[INFO 03-02 09:13:23] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(node_name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, model_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(node_name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, model_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 03-02 09:13:23] ax.api.client: Generated new trial 0 with parameters {'base_bid': 60.0, 'bid_to_pay_ratio': 0.5, 'ctr_coefficient': 0.5, 'pacing_coefficient': 0.5} using GenerationNode CenterOfSearchSpace.

Simulating date: 20130607 with base_bid: 60.0, bid_to_pay_ratio: 0.5, ctr_coefficient: 0.5, pacing_coefficient: 0.5

[INFO 03-02 09:13:36] ax.api.client: Trial 0 marked COMPLETED.
[INFO 03-02 09:13:36] ax.api.client: Generated new trial 1 with parameters {'base_bid': 70.95345, 'bid_to_pay_ratio': 0.806773, 'ctr_coefficient': 0.946587, 'pacing_coefficient': 0.501761} using GenerationNode Sobol.

Total clicks: 66, impressions: 101882, budget spent: 6019811.0
Simulating date: 20130608 with base_bid: 70.9534502029419, bid_to_pay_ratio: 0.8067728877067566, ctr_coefficient: 0.946586549282074, pacing_coefficient: 0.5017608404159546

[INFO 03-02 09:13:46] ax.api.client: Trial 1 marked COMPLETED.
[INFO 03-02 09:13:46] ax.api.client: Generated new trial 2 with parameters {'base_bid': 42.848856, 'bid_to_pay_ratio': 0.109523, 'ctr_coefficient': 0.021564, 'pacing_coefficient': 0.278944} using GenerationNode Sobol.

Total clicks: 71, impressions: 82758, budget spent: 6019819.0
Simulating date: 20130609 with base_bid: 42.84885585308075, bid_to_pay_ratio: 0.10952319391071796, ctr_coefficient: 0.021563850343227386, pacing_coefficient: 0.2789439717307687

[INFO 03-02 09:14:37] ax.api.client: Trial 2 marked COMPLETED.
[INFO 03-02 09:14:37] ax.api.client: Generated new trial 3 with parameters {'base_bid': 38.319603, 'bid_to_pay_ratio': 0.527505, 'ctr_coefficient': 0.749896, 'pacing_coefficient': 0.982417} using GenerationNode Sobol.

Total clicks: 104, impressions: 171964, budget spent: 5165425.0
Simulating date: 20130610 with base_bid: 38.31960268318653, bid_to_pay_ratio: 0.5275049116462469, ctr_coefficient: 0.7498957207426429, pacing_coefficient: 0.9824169147759676

[INFO 03-02 09:14:49] ax.api.client: Trial 3 marked COMPLETED.
[INFO 03-02 09:14:49] ax.api.client: Generated new trial 4 with parameters {'base_bid': 85.175418, 'bid_to_pay_ratio': 0.334161, 'ctr_coefficient': 0.281973, 'pacing_coefficient': 0.235902} using GenerationNode Sobol.

Total clicks: 75, impressions: 94354, budget spent: 6019791.0
Simulating date: 20130611 with base_bid: 85.17541788518429, bid_to_pay_ratio: 0.3341609966009855, ctr_coefficient: 0.2819729968905449, pacing_coefficient: 0.2359022032469511

[INFO 03-02 09:15:01] ax.api.client: Trial 4 marked COMPLETED.

Total clicks: 90, impressions: 97611, budget spent: 6019818.0

[INFO 03-02 09:15:02] ax.api.client: Generated new trial 5 with parameters {'base_bid': 76.22551, 'bid_to_pay_ratio': 0.038676, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.049357} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 76.22550973676135, bid_to_pay_ratio: 0.038675797429113014, ctr_coefficient: 0.0, pacing_coefficient: 0.04935731579017902

[INFO 03-02 09:15:22] ax.api.client: Trial 5 marked COMPLETED.

Total clicks: 93, impressions: 127069, budget spent: 6019779.0

[INFO 03-02 09:15:22] ax.api.client: Generated new trial 6 with parameters {'base_bid': 68.915002, 'bid_to_pay_ratio': 0.002363, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.429738} using GenerationNode MBM.

Simulating date: 20130607 with base_bid: 68.91500229026869, bid_to_pay_ratio: 0.0023632406550437414, ctr_coefficient: 0.0, pacing_coefficient: 0.4297384347490085

[INFO 03-02 09:15:43] ax.api.client: Trial 6 marked COMPLETED.

Total clicks: 78, impressions: 139009, budget spent: 6019818.0

[INFO 03-02 09:15:43] ax.api.client: Generated new trial 7 with parameters {'base_bid': 33.797365, 'bid_to_pay_ratio': 0.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.204402} using GenerationNode MBM.

Simulating date: 20130608 with base_bid: 33.79736548581379, bid_to_pay_ratio: 0.0, ctr_coefficient: 0.0, pacing_coefficient: 0.20440237453839027

[INFO 03-02 09:16:28] ax.api.client: Trial 7 marked COMPLETED.

Total clicks: 59, impressions: 118213, budget spent: 2429187.0

[INFO 03-02 09:16:28] ax.api.client: Generated new trial 8 with parameters {'base_bid': 65.960234, 'bid_to_pay_ratio': 0.107763, 'ctr_coefficient': 0.01723, 'pacing_coefficient': 0.198373} using GenerationNode MBM.

Simulating date: 20130609 with base_bid: 65.9602343254001, bid_to_pay_ratio: 0.10776286145517365, ctr_coefficient: 0.01722996745168017, pacing_coefficient: 0.19837330775492892

[INFO 03-02 09:16:51] ax.api.client: Trial 8 marked COMPLETED.

Total clicks: 90, impressions: 135790, budget spent: 6019786.0

[INFO 03-02 09:16:51] ax.api.client: Generated new trial 9 with parameters {'base_bid': 31.463362, 'bid_to_pay_ratio': 0.127327, 'ctr_coefficient': 0.026717, 'pacing_coefficient': 0.303248} using GenerationNode MBM.

Simulating date: 20130610 with base_bid: 31.463362262559237, bid_to_pay_ratio: 0.12732697549524924, ctr_coefficient: 0.026717200494175995, pacing_coefficient: 0.30324760290620295

[INFO 03-02 09:17:38] ax.api.client: Trial 9 marked COMPLETED.

Total clicks: 68, impressions: 122199, budget spent: 2874361.0

[INFO 03-02 09:17:39] ax.api.client: Generated new trial 10 with parameters {'base_bid': 45.045172, 'bid_to_pay_ratio': 0.13484, 'ctr_coefficient': 0.022876, 'pacing_coefficient': 0.294161} using GenerationNode MBM.

Simulating date: 20130611 with base_bid: 45.04517161046832, bid_to_pay_ratio: 0.13483961854615642, ctr_coefficient: 0.022876441745184942, pacing_coefficient: 0.29416121173686727

[INFO 03-02 09:18:24] ax.api.client: Trial 10 marked COMPLETED.

Total clicks: 119, impressions: 183277, budget spent: 6019800.0

[INFO 03-02 09:18:24] ax.api.client: Generated new trial 11 with parameters {'base_bid': 47.325359, 'bid_to_pay_ratio': 0.159281, 'ctr_coefficient': 0.018827, 'pacing_coefficient': 0.308915} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 47.32535867695698, bid_to_pay_ratio: 0.15928149015303264, ctr_coefficient: 0.01882693067994852, pacing_coefficient: 0.30891475646771954

[INFO 03-02 09:19:08] ax.api.client: Trial 11 marked COMPLETED.

Total clicks: 106, impressions: 186446, budget spent: 6019813.0

[INFO 03-02 09:19:09] ax.api.client: Generated new trial 12 with parameters {'base_bid': 45.177082, 'bid_to_pay_ratio': 0.170616, 'ctr_coefficient': 0.225428, 'pacing_coefficient': 0.241836} using GenerationNode MBM.

Simulating date: 20130607 with base_bid: 45.17708188633905, bid_to_pay_ratio: 0.17061581401194145, ctr_coefficient: 0.22542761281044696, pacing_coefficient: 0.24183571460172196

[INFO 03-02 09:19:28] ax.api.client: Trial 12 marked COMPLETED.

Total clicks: 82, impressions: 133313, budget spent: 6019779.0

[INFO 03-02 09:19:28] ax.api.client: Generated new trial 13 with parameters {'base_bid': 47.638525, 'bid_to_pay_ratio': 0.126786, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.315732} using GenerationNode MBM.

Simulating date: 20130608 with base_bid: 47.638524668886525, bid_to_pay_ratio: 0.1267858603785328, ctr_coefficient: 0.0, pacing_coefficient: 0.31573155018369925

[INFO 03-02 09:20:13] ax.api.client: Trial 13 marked COMPLETED.

Total clicks: 126, impressions: 171388, budget spent: 5167237.0

[INFO 03-02 09:20:14] ax.api.client: Generated new trial 14 with parameters {'base_bid': 50.752059, 'bid_to_pay_ratio': 0.128913, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.185817} using GenerationNode MBM.

Simulating date: 20130609 with base_bid: 50.75205874759392, bid_to_pay_ratio: 0.1289132526819515, ctr_coefficient: 0.0, pacing_coefficient: 0.1858165288447184

[INFO 03-02 09:21:00] ax.api.client: Trial 14 marked COMPLETED.

Total clicks: 117, impressions: 181839, budget spent: 5692592.0

[INFO 03-02 09:21:00] ax.api.client: Generated new trial 15 with parameters {'base_bid': 49.679018, 'bid_to_pay_ratio': 0.128708, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.438042} using GenerationNode MBM.

Simulating date: 20130610 with base_bid: 49.67901803565202, bid_to_pay_ratio: 0.12870787951241638, ctr_coefficient: 0.0, pacing_coefficient: 0.43804242777284946

[INFO 03-02 09:21:34] ax.api.client: Trial 15 marked COMPLETED.

Total clicks: 86, impressions: 157555, budget spent: 6019797.0

[INFO 03-02 09:21:35] ax.api.client: Generated new trial 16 with parameters {'base_bid': 50.607586, 'bid_to_pay_ratio': 0.126312, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.27571} using GenerationNode MBM.

Simulating date: 20130611 with base_bid: 50.60758601079947, bid_to_pay_ratio: 0.12631192691868812, ctr_coefficient: 0.0, pacing_coefficient: 0.27570977146313186

[INFO 03-02 09:22:16] ax.api.client: Trial 16 marked COMPLETED.

Total clicks: 107, impressions: 177058, budget spent: 6019805.0

[INFO 03-02 09:22:17] ax.api.client: Generated new trial 17 with parameters {'base_bid': 47.351188, 'bid_to_pay_ratio': 0.116695, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.211353} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 47.351188331897525, bid_to_pay_ratio: 0.1166948604820313, ctr_coefficient: 0.0, pacing_coefficient: 0.21135325170382924

[INFO 03-02 09:23:05] ax.api.client: Trial 17 marked COMPLETED.

Total clicks: 94, impressions: 184941, budget spent: 5433277.0

best_parameters, prediction, index, name = client.get_best_parameterization()
print("Best Parameters:", best_parameters)
print("Prediction (mean, variance):", prediction)

Best Parameters: {'base_bid': 47.638524668886525, 'bid_to_pay_ratio': 0.1267858603785328, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.31573155018369925}
Prediction (mean, variance): {'clicks': (np.float64(125.28789641416625), np.float64(2.4957847821450057))}

Surprisingly, the best result was obtained when the base bid was lower and bid to pay ratio was leveraged.

In the manual search for the best hyperparameters we reached a maximum of 103 clicks. Using Bayesian Optimization, we reached an experimental result of 126 clicks.

from ax.analysis.plotly.surface.slice import SlicePlot

analysis = SlicePlot("base_bid")

analysis.compute(
    experiment=client._experiment,
    generation_strategy=client._generation_strategy,
    adapter=None,
)

This plot shows that, for lower base bids, the number of clicks has a lot of uncertainty, because it’s very dependent on the other parameters.

from ax.analysis.plotly.arm_effects import ArmEffectsPlot

analysis = ArmEffectsPlot()

analysis.compute(
    experiment=client._experiment,
    generation_strategy=client._generation_strategy,
    adapter=None,
)

2.4.5.2 Maximize the number of impressions¶

Assuming we want to maximize the number of impressions. The number of impressions is much higher, so we can use shorter time periods for the trials.

I’ll run experiments of 10K bids each.

# In this case we make the budget 10x lower as the trials are for 100K bids, much smaller. The budget is not per day now.
budget = 601977.6
trial_size = 100000

client = Client()

parameters = [
    RangeParameterConfig(
        name="base_bid", parameter_type="float", bounds=(20, 500)
    ),
    RangeParameterConfig(
        name="bid_to_pay_ratio", parameter_type="float", bounds=(0, 1)
    ),
    RangeParameterConfig(
        name="ctr_coefficient", parameter_type="float", bounds=(0, 1)
    ),
    RangeParameterConfig(
        name="pacing_coefficient", parameter_type="float", bounds=(0, 1)
    ),
]

client.configure_experiment(parameters=parameters)
client.configure_optimization(objective="impressions")

for date in [
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
]:
    offset = 0
    while True:
        trials = client.get_next_trials(max_trials=1)
        trial_index = list(trials.keys())[0]
        parameters = trials[trial_index]

        base_bid = parameters["base_bid"]
        bid_to_pay_ratio = parameters["bid_to_pay_ratio"]
        ctr_coefficient = parameters["ctr_coefficient"]
        pacing_coefficient = parameters["pacing_coefficient"]

        print(
            f"Simulating date: {date}, offset: {offset} with base_bid: {base_bid}, bid_to_pay_ratio: {bid_to_pay_ratio}, ctr_coefficient: {ctr_coefficient}, pacing_coefficient: {pacing_coefficient}"
        )
        (
            total_bids,
            total_bids_done,
            total_bids_won,
            total_budget_spent,
            total_clicks,
        ) = dsp_simulation(
            date,
            base_bid,
            budget,
            bid_to_pay_ratio,
            ctr_coefficient,
            pacing_coefficient,
            offset=offset,
            limit=trial_size,
        )
        offset += trial_size
        if total_bids <= 0:
            break
        print(
            f"Total clicks: {total_clicks}, impressions: {total_bids_won}, budget spent: {total_budget_spent}"
        )

        client.complete_trial(
            trial_index=trial_index,
            raw_data={
                "clicks": total_clicks,
                "impressions": total_bids_won,
                "budget_spent": total_budget_spent,
            },
        )

[INFO 03-02 09:23:07] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(node_name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, model_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(node_name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, model_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 03-02 09:23:07] ax.api.client: Generated new trial 0 with parameters {'base_bid': 260.0, 'bid_to_pay_ratio': 0.5, 'ctr_coefficient': 0.5, 'pacing_coefficient': 0.5} using GenerationNode CenterOfSearchSpace.

Simulating date: 20130607, offset: 0 with base_bid: 260.0, bid_to_pay_ratio: 0.5, ctr_coefficient: 0.5, pacing_coefficient: 0.5

[INFO 03-02 09:23:08] ax.api.client: Trial 0 marked COMPLETED.
[INFO 03-02 09:23:08] ax.api.client: Generated new trial 1 with parameters {'base_bid': 240.380664, 'bid_to_pay_ratio': 0.57677, 'ctr_coefficient': 0.558877, 'pacing_coefficient': 0.093868} using GenerationNode Sobol.

Total clicks: 11, impressions: 8656, budget spent: 602051.0
Simulating date: 20130607, offset: 100000 with base_bid: 240.38066387176514, bid_to_pay_ratio: 0.5767697691917419, ctr_coefficient: 0.5588765144348145, pacing_coefficient: 0.09386793524026871

[INFO 03-02 09:23:10] ax.api.client: Trial 1 marked COMPLETED.
[INFO 03-02 09:23:10] ax.api.client: Generated new trial 2 with parameters {'base_bid': 385.61802, 'bid_to_pay_ratio': 0.032145, 'ctr_coefficient': 0.04485, 'pacing_coefficient': 0.824499} using GenerationNode Sobol.

Total clicks: 4, impressions: 8982, budget spent: 602050.0
Simulating date: 20130607, offset: 200000 with base_bid: 385.6180199980736, bid_to_pay_ratio: 0.032144815661013126, ctr_coefficient: 0.04484965931624174, pacing_coefficient: 0.8244987512007356

[INFO 03-02 09:23:11] ax.api.client: Trial 2 marked COMPLETED.
[INFO 03-02 09:23:11] ax.api.client: Generated new trial 3 with parameters {'base_bid': 320.42189, 'bid_to_pay_ratio': 0.817924, 'ctr_coefficient': 0.938358, 'pacing_coefficient': 0.297223} using GenerationNode Sobol.

Total clicks: 6, impressions: 9546, budget spent: 602144.0
Simulating date: 20130607, offset: 300000 with base_bid: 320.4218904674053, bid_to_pay_ratio: 0.8179240869358182, ctr_coefficient: 0.9383577965199947, pacing_coefficient: 0.2972226496785879

[INFO 03-02 09:23:13] ax.api.client: Trial 3 marked COMPLETED.
[INFO 03-02 09:23:13] ax.api.client: Generated new trial 4 with parameters {'base_bid': 63.709801, 'bid_to_pay_ratio': 0.292953, 'ctr_coefficient': 0.454376, 'pacing_coefficient': 0.527392} using GenerationNode Sobol.

Total clicks: 8, impressions: 10332, budget spent: 602039.0
Simulating date: 20130607, offset: 400000 with base_bid: 63.70980113744736, bid_to_pay_ratio: 0.29295273311436176, ctr_coefficient: 0.45437593199312687, pacing_coefficient: 0.5273921256884933

[INFO 03-02 09:23:14] ax.api.client: Trial 4 marked COMPLETED.

Total clicks: 10, impressions: 10418, budget spent: 601985.0

[INFO 03-02 09:23:15] ax.api.client: Generated new trial 5 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.009414, 'ctr_coefficient': 0.439046, 'pacing_coefficient': 0.136876} using GenerationNode MBM.

Simulating date: 20130607, offset: 500000 with base_bid: 20.0, bid_to_pay_ratio: 0.009414296117452911, ctr_coefficient: 0.4390463462546973, pacing_coefficient: 0.13687571722436787

[INFO 03-02 09:23:16] ax.api.client: Generated new trial 6 with parameters {'base_bid': 26.363137, 'bid_to_pay_ratio': 0.602159, 'ctr_coefficient': 0.469386, 'pacing_coefficient': 0.977226} using GenerationNode MBM.

Simulating date: 20130608, offset: 0 with base_bid: 26.363136505747285, bid_to_pay_ratio: 0.602158844724566, ctr_coefficient: 0.46938603559716935, pacing_coefficient: 0.977225896610449

[INFO 03-02 09:23:18] ax.api.client: Trial 6 marked COMPLETED.

Total clicks: 11, impressions: 10713, budget spent: 602021.0

[INFO 03-02 09:23:18] ax.api.client: Generated new trial 7 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.867429, 'ctr_coefficient': 0.707441, 'pacing_coefficient': 0.677151} using GenerationNode MBM.

Simulating date: 20130608, offset: 100000 with base_bid: 20.0, bid_to_pay_ratio: 0.8674286901365595, ctr_coefficient: 0.7074411451939768, pacing_coefficient: 0.6771507716044439

[INFO 03-02 09:23:20] ax.api.client: Trial 7 marked COMPLETED.

Total clicks: 7, impressions: 11421, budget spent: 602000.0

[INFO 03-02 09:23:21] ax.api.client: Generated new trial 8 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.846753, 'ctr_coefficient': 0.966287, 'pacing_coefficient': 0.155962} using GenerationNode MBM.

Simulating date: 20130608, offset: 200000 with base_bid: 20.0, bid_to_pay_ratio: 0.8467532397861516, ctr_coefficient: 0.9662865183681965, pacing_coefficient: 0.15596162606757466

[INFO 03-02 09:23:24] ax.api.client: Trial 8 marked COMPLETED.

Total clicks: 7, impressions: 11620, budget spent: 602033.0

[INFO 03-02 09:23:24] ax.api.client: Generated new trial 9 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.0, 'ctr_coefficient': 0.975432, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130608, offset: 300000 with base_bid: 20.0, bid_to_pay_ratio: 0.0, ctr_coefficient: 0.9754316363233696, pacing_coefficient: 1.0

[INFO 03-02 09:23:26] ax.api.client: Trial 9 marked COMPLETED.

Total clicks: 9, impressions: 9866, budget spent: 601979.0

[INFO 03-02 09:23:26] ax.api.client: Generated new trial 10 with parameters {'base_bid': 52.003716, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.913121, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130608, offset: 400000 with base_bid: 52.00371558789789, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.9131214308057066, pacing_coefficient: 0.0

[INFO 03-02 09:23:28] ax.api.client: Trial 10 marked COMPLETED.

Total clicks: 8, impressions: 9307, budget spent: 602001.0

[INFO 03-02 09:23:28] ax.api.client: Generated new trial 11 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.787791, 'ctr_coefficient': 0.771205, 'pacing_coefficient': 0.586501} using GenerationNode MBM.

Simulating date: 20130608, offset: 500000 with base_bid: 20.0, bid_to_pay_ratio: 0.7877908149615163, ctr_coefficient: 0.7712053499105103, pacing_coefficient: 0.5865010149165992

[INFO 03-02 09:23:29] ax.api.client: Generated new trial 12 with parameters {'base_bid': 85.054532, 'bid_to_pay_ratio': 0.815079, 'ctr_coefficient': 1.0, 'pacing_coefficient': 0.512106} using GenerationNode MBM.

Simulating date: 20130609, offset: 0 with base_bid: 85.05453167551755, bid_to_pay_ratio: 0.8150786924707945, ctr_coefficient: 1.0, pacing_coefficient: 0.5121056654005423

[INFO 03-02 09:23:31] ax.api.client: Trial 12 marked COMPLETED.

Total clicks: 5, impressions: 8599, budget spent: 601999.0

[INFO 03-02 09:23:31] ax.api.client: Generated new trial 13 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.858595, 'ctr_coefficient': 0.569357, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609, offset: 100000 with base_bid: 20.0, bid_to_pay_ratio: 0.8585947686182827, ctr_coefficient: 0.5693568298583098, pacing_coefficient: 0.0

[INFO 03-02 09:23:37] ax.api.client: Trial 13 marked COMPLETED.

Total clicks: 12, impressions: 18363, budget spent: 601991.0

[INFO 03-02 09:23:38] ax.api.client: Generated new trial 14 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.963805, 'ctr_coefficient': 0.532031, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609, offset: 200000 with base_bid: 20.0, bid_to_pay_ratio: 0.9638045359016248, ctr_coefficient: 0.532031322674213, pacing_coefficient: 0.0

[INFO 03-02 09:23:44] ax.api.client: Trial 14 marked COMPLETED.

Total clicks: 11, impressions: 18637, budget spent: 601985.0

[INFO 03-02 09:23:45] ax.api.client: Generated new trial 15 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.919618, 'ctr_coefficient': 0.38707, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609, offset: 300000 with base_bid: 20.0, bid_to_pay_ratio: 0.9196179598123673, ctr_coefficient: 0.3870703305868475, pacing_coefficient: 0.0

[INFO 03-02 09:23:54] ax.api.client: Trial 15 marked COMPLETED.

Total clicks: 23, impressions: 19762, budget spent: 601981.0

[INFO 03-02 09:23:55] ax.api.client: Generated new trial 16 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.152024, 'ctr_coefficient': 0.287686, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609, offset: 400000 with base_bid: 20.0, bid_to_pay_ratio: 0.1520235799531176, ctr_coefficient: 0.28768594725404456, pacing_coefficient: 0.0

[INFO 03-02 09:23:58] ax.api.client: Trial 16 marked COMPLETED.

Total clicks: 5, impressions: 5573, budget spent: 126613.0

[INFO 03-02 09:23:59] ax.api.client: Generated new trial 17 with parameters {'base_bid': 56.590123, 'bid_to_pay_ratio': 0.936587, 'ctr_coefficient': 0.32373, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609, offset: 500000 with base_bid: 56.59012330608486, bid_to_pay_ratio: 0.936587050652247, ctr_coefficient: 0.3237296427948197, pacing_coefficient: 0.0

[INFO 03-02 09:24:00] ax.api.client: Generated new trial 18 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.310076, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 0 with base_bid: 20.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.3100756234624572, pacing_coefficient: 0.0

[INFO 03-02 09:24:11] ax.api.client: Trial 18 marked COMPLETED.

Total clicks: 19, impressions: 21033, budget spent: 602043.0

[INFO 03-02 09:24:11] ax.api.client: Generated new trial 19 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.130068, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 100000 with base_bid: 20.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.1300680071150948, pacing_coefficient: 0.0

[INFO 03-02 09:24:22] ax.api.client: Trial 19 marked COMPLETED.

Total clicks: 10, impressions: 11783, budget spent: 285814.0

[INFO 03-02 09:24:23] ax.api.client: Generated new trial 20 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.327955, 'pacing_coefficient': 0.318335} using GenerationNode MBM.

Simulating date: 20130610, offset: 200000 with base_bid: 20.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.32795476981325994, pacing_coefficient: 0.31833508466384897

[INFO 03-02 09:24:30] ax.api.client: Trial 20 marked COMPLETED.

Total clicks: 10, impressions: 17620, budget spent: 601987.0

[INFO 03-02 09:24:30] ax.api.client: Generated new trial 21 with parameters {'base_bid': 67.709064, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.344275, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 300000 with base_bid: 67.70906434763414, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.3442753015590231, pacing_coefficient: 0.0

[INFO 03-02 09:24:32] ax.api.client: Trial 21 marked COMPLETED.

Total clicks: 5, impressions: 11245, budget spent: 601998.0

[INFO 03-02 09:24:33] ax.api.client: Generated new trial 22 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.348769, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 400000 with base_bid: 20.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.34876865794836914, pacing_coefficient: 0.0

[INFO 03-02 09:24:37] ax.api.client: Trial 22 marked COMPLETED.

Total clicks: 11, impressions: 10355, budget spent: 316921.0

[INFO 03-02 09:24:38] ax.api.client: Generated new trial 23 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.771782, 'ctr_coefficient': 0.438259, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 500000 with base_bid: 20.0, bid_to_pay_ratio: 0.771782232342218, ctr_coefficient: 0.4382592939854129, pacing_coefficient: 0.0

[INFO 03-02 09:24:39] ax.api.client: Generated new trial 24 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.845303, 'ctr_coefficient': 0.388208, 'pacing_coefficient': 0.288141} using GenerationNode MBM.

Simulating date: 20130611, offset: 0 with base_bid: 20.0, bid_to_pay_ratio: 0.8453034601563649, ctr_coefficient: 0.3882076364578668, pacing_coefficient: 0.288141420135192

[INFO 03-02 09:24:44] ax.api.client: Trial 24 marked COMPLETED.

Total clicks: 17, impressions: 17582, budget spent: 601988.0

[INFO 03-02 09:24:44] ax.api.client: Generated new trial 25 with parameters {'base_bid': 146.76127, 'bid_to_pay_ratio': 0.0, 'ctr_coefficient': 0.739221, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 100000 with base_bid: 146.76127043657206, bid_to_pay_ratio: 0.0, ctr_coefficient: 0.7392212407723535, pacing_coefficient: 1.0

[INFO 03-02 09:24:46] ax.api.client: Trial 25 marked COMPLETED.

Total clicks: 4, impressions: 8731, budget spent: 601996.0

[INFO 03-02 09:24:46] ax.api.client: Generated new trial 26 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.950523, 'ctr_coefficient': 0.162256, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 200000 with base_bid: 20.0, bid_to_pay_ratio: 0.9505233594814887, ctr_coefficient: 0.1622559230041762, pacing_coefficient: 1.0

[INFO 03-02 09:24:55] ax.api.client: Trial 26 marked COMPLETED.

Total clicks: 14, impressions: 18104, budget spent: 601983.0

[INFO 03-02 09:24:55] ax.api.client: Generated new trial 27 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.764617, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 300000 with base_bid: 20.0, bid_to_pay_ratio: 0.7646165472792839, ctr_coefficient: 0.0, pacing_coefficient: 1.0

[INFO 03-02 09:25:07] ax.api.client: Trial 27 marked COMPLETED.

Total clicks: 24, impressions: 19743, budget spent: 439691.0

[INFO 03-02 09:25:07] ax.api.client: Generated new trial 28 with parameters {'base_bid': 63.638832, 'bid_to_pay_ratio': 0.803713, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 400000 with base_bid: 63.63883183228494, bid_to_pay_ratio: 0.8037128176079494, ctr_coefficient: 0.0, pacing_coefficient: 1.0

[INFO 03-02 09:25:10] ax.api.client: Trial 28 marked COMPLETED.

Total clicks: 14, impressions: 14189, budget spent: 601999.0

[INFO 03-02 09:25:11] ax.api.client: Generated new trial 29 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.757961, 'ctr_coefficient': 0.143448, 'pacing_coefficient': 0.49852} using GenerationNode MBM.

Simulating date: 20130611, offset: 500000 with base_bid: 20.0, bid_to_pay_ratio: 0.7579613432630203, ctr_coefficient: 0.1434479048451166, pacing_coefficient: 0.4985201364651934

[INFO 03-02 09:25:12] ax.api.client: Generated new trial 30 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.957947, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 0 with base_bid: 20.0, bid_to_pay_ratio: 0.9579469695937557, ctr_coefficient: 0.0, pacing_coefficient: 1.0

[INFO 03-02 09:25:23] ax.api.client: Trial 30 marked COMPLETED.

Total clicks: 21, impressions: 20821, budget spent: 502325.0

[INFO 03-02 09:25:24] ax.api.client: Generated new trial 31 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 100000 with base_bid: 20.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0, pacing_coefficient: 1.0

[INFO 03-02 09:25:35] ax.api.client: Trial 31 marked COMPLETED.

Total clicks: 9, impressions: 17972, budget spent: 460634.0

[INFO 03-02 09:25:35] ax.api.client: Generated new trial 32 with parameters {'base_bid': 465.182325, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 1.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 200000 with base_bid: 465.1823254220783, bid_to_pay_ratio: 1.0, ctr_coefficient: 1.0, pacing_coefficient: 0.0

[INFO 03-02 09:25:37] ax.api.client: Trial 32 marked COMPLETED.

Total clicks: 13, impressions: 8948, budget spent: 602038.0

[INFO 03-02 09:25:38] ax.api.client: Generated new trial 33 with parameters {'base_bid': 500.0, 'bid_to_pay_ratio': 0.0, 'ctr_coefficient': 1.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 300000 with base_bid: 500.0, bid_to_pay_ratio: 0.0, ctr_coefficient: 1.0, pacing_coefficient: 1.0

[INFO 03-02 09:25:39] ax.api.client: Trial 33 marked COMPLETED.

Total clicks: 9, impressions: 8214, budget spent: 602003.0

[INFO 03-02 09:25:40] ax.api.client: Generated new trial 34 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 0.876962, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.70516} using GenerationNode MBM.

Simulating date: 20130612, offset: 400000 with base_bid: 20.0, bid_to_pay_ratio: 0.8769619328884315, ctr_coefficient: 0.0, pacing_coefficient: 0.7051599500301483

[INFO 03-02 09:25:45] ax.api.client: Trial 34 marked COMPLETED.

Total clicks: 4, impressions: 11191, budget spent: 226090.0

[INFO 03-02 09:25:46] ax.api.client: Generated new trial 35 with parameters {'base_bid': 20.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.533363, 'pacing_coefficient': 0.204942} using GenerationNode MBM.

Simulating date: 20130612, offset: 500000 with base_bid: 20.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.5333634623278097, pacing_coefficient: 0.2049419383693365

Here we’re trying to maximize the number of impressions with a much more limited budget.

If the bid is too low, we risk losing the auction. If it’s too high, the budget will be consumed faster in expensive bids.

Note: To maximize the impressions we should have use a different formula, predicting the CTR will take us to bet on potentially better impressions, but not to increase the number of impressions.

best_parameters, prediction, index, name = client.get_best_parameterization()
print("Best Parameters:", best_parameters)
print("Prediction (mean, variance):", prediction)

Best Parameters: {'base_bid': 20.0, 'bid_to_pay_ratio': 0.9579469695937557, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0}
Prediction (mean, variance): {'impressions': (np.float64(18608.05927906453), np.float64(1412819.9050824668))}

from ax.analysis.plotly.arm_effects import ArmEffectsPlot

analysis = ArmEffectsPlot()

analysis.compute(
    experiment=client._experiment,
    generation_strategy=client._generation_strategy,
    adapter=None,
)

from ax.analysis.plotly.surface.slice import SlicePlot

analysis = SlicePlot("base_bid")

analysis.compute(
    experiment=client._experiment,
    generation_strategy=client._generation_strategy,
    adapter=None,
)

We can see how the number of impressions went up when it started to try lower base bids, opposite to what I expected.

It can also be seen that other coefficients are low, because they don’t provide an advantage to maximize the number of impressions.

2.4.6. Multi-Objective Optimization¶

2.4.6.1. Maximize the number of clicks while minimizing the budget spent¶

All the previous experiments run with a fixed budget. What happens if we want to decide the budget and see how the budget increase would generate a growth in clicks (Or not).

Running multi-objective optimization requires much more trials, as we have two dimensions to explore. For this reason, and given that we only have 6 days of data, we’ll do a rolling experiment and start again from the first day. The experiment will run for 30 days.

# Now the budget should be set to a number that doesn't limit the bids. We still have to provide one.
budget = 100000000000.0
# In this case the pacing_coefficient should be 0 because the budget is already too large.

client = Client()

parameters = [
    RangeParameterConfig(
        name="base_bid", parameter_type="float", bounds=(1, 100)
    ),  # The minimum bid is lowered too to allow budget reduction
    RangeParameterConfig(
        name="bid_to_pay_ratio", parameter_type="float", bounds=(0, 1)
    ),
    RangeParameterConfig(
        name="ctr_coefficient", parameter_type="float", bounds=(0, 1)
    ),
    RangeParameterConfig(
        name="pacing_coefficient", parameter_type="float", bounds=(0, 1)
    ),
]

client.configure_experiment(parameters=parameters)
client.configure_optimization(
    objective="-budget_spent, clicks",
    outcome_constraints=["budget_spent <= 6009888.0", "clicks >= 1"],
)

for date in [
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
]:
    trials = client.get_next_trials(max_trials=1)
    trial_index = list(trials.keys())[0]
    parameters = trials[trial_index]

    base_bid = parameters["base_bid"]
    bid_to_pay_ratio = parameters["bid_to_pay_ratio"]
    ctr_coefficient = parameters["ctr_coefficient"]

    print(
        f"Simulating date: {date} with base_bid: {base_bid}, bid_to_pay_ratio: {bid_to_pay_ratio}, ctr_coefficient: {ctr_coefficient}"
    )
    (
        total_bids,
        total_bids_done,
        total_bids_won,
        total_budget_spent,
        total_clicks,
    ) = dsp_simulation(
        date, base_bid, budget, bid_to_pay_ratio, ctr_coefficient, 0
    )
    print(
        f"Total clicks: {total_clicks}, impressions: {total_bids_won}, budget spent: {total_budget_spent}"
    )

    client.complete_trial(
        trial_index=trial_index,
        raw_data={
            "clicks": total_clicks,
            "impressions": total_bids_won,
            "budget_spent": total_budget_spent,
        },
    )

[INFO 03-02 09:25:48] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(node_name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, model_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(node_name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, model_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 03-02 09:25:48] ax.api.client: Generated new trial 0 with parameters {'base_bid': 50.5, 'bid_to_pay_ratio': 0.5, 'ctr_coefficient': 0.5, 'pacing_coefficient': 0.5} using GenerationNode CenterOfSearchSpace.

Simulating date: 20130607 with base_bid: 50.5, bid_to_pay_ratio: 0.5, ctr_coefficient: 0.5

[INFO 03-02 09:26:41] ax.api.client: Trial 0 marked COMPLETED.
[INFO 03-02 09:26:41] ax.api.client: Generated new trial 1 with parameters {'base_bid': 64.481851, 'bid_to_pay_ratio': 0.806906, 'ctr_coefficient': 0.952859, 'pacing_coefficient': 0.986439} using GenerationNode Sobol.

Total clicks: 277, impressions: 430327, budget spent: 21461926.0
Simulating date: 20130608 with base_bid: 64.48185133934021, bid_to_pay_ratio: 0.8069061040878296, ctr_coefficient: 0.9528588652610779

[INFO 03-02 09:27:25] ax.api.client: Trial 1 marked COMPLETED.
[INFO 03-02 09:27:25] ax.api.client: Generated new trial 2 with parameters {'base_bid': 10.188729, 'bid_to_pay_ratio': 0.189532, 'ctr_coefficient': 0.357067, 'pacing_coefficient': 0.456972} using GenerationNode Sobol.

Total clicks: 354, impressions: 397213, budget spent: 27095224.0
Simulating date: 20130609 with base_bid: 10.188729404471815, bid_to_pay_ratio: 0.189531821757555, ctr_coefficient: 0.35706725530326366

[INFO 03-02 09:28:10] ax.api.client: Trial 2 marked COMPLETED.
[INFO 03-02 09:28:10] ax.api.client: Generated new trial 3 with parameters {'base_bid': 46.043281, 'bid_to_pay_ratio': 0.536623, 'ctr_coefficient': 0.601897, 'pacing_coefficient': 0.67317} using GenerationNode Sobol.

Total clicks: 30, impressions: 76402, budget spent: 1163053.0
Simulating date: 20130610 with base_bid: 46.0432806443423, bid_to_pay_ratio: 0.5366230830550194, ctr_coefficient: 0.6018973160535097

[INFO 03-02 09:28:55] ax.api.client: Trial 3 marked COMPLETED.
[INFO 03-02 09:28:55] ax.api.client: Generated new trial 4 with parameters {'base_bid': 78.193875, 'bid_to_pay_ratio': 0.46662, 'ctr_coefficient': 0.197567, 'pacing_coefficient': 0.14135} using GenerationNode Sobol.

Total clicks: 331, impressions: 383995, budget spent: 21563691.0
Simulating date: 20130611 with base_bid: 78.19387516751885, bid_to_pay_ratio: 0.46662039402872324, ctr_coefficient: 0.19756684452295303

[INFO 03-02 09:29:42] ax.api.client: Trial 4 marked COMPLETED.

Total clicks: 341, impressions: 383277, budget spent: 21016420.0

[INFO 03-02 09:29:43] ax.api.client: Generated new trial 5 with parameters {'base_bid': 1.0, 'bid_to_pay_ratio': 0.059797, 'ctr_coefficient': 0.959013, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 1.0, bid_to_pay_ratio: 0.059796565289867425, ctr_coefficient: 0.9590131692616928

[INFO 03-02 09:30:31] ax.api.client: Trial 5 marked COMPLETED.

Total clicks: 0, impressions: 2121, budget spent: 8571.0

[INFO 03-02 09:30:32] ax.api.client: Generated new trial 6 with parameters {'base_bid': 1.0, 'bid_to_pay_ratio': 0.555927, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130607 with base_bid: 1.0, bid_to_pay_ratio: 0.5559273750621235, ctr_coefficient: 0.0

[INFO 03-02 09:31:21] ax.api.client: Trial 6 marked COMPLETED.

Total clicks: 0, impressions: 23, budget spent: 109.0

[INFO 03-02 09:31:22] ax.api.client: Generated new trial 7 with parameters {'base_bid': 1.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130608 with base_bid: 1.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0

[INFO 03-02 09:32:06] ax.api.client: Trial 7 marked COMPLETED.

Total clicks: 0, impressions: 37, budget spent: 892.0

[INFO 03-02 09:32:07] ax.api.client: Generated new trial 8 with parameters {'base_bid': 17.847489, 'bid_to_pay_ratio': 0.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130609 with base_bid: 17.847489495413697, bid_to_pay_ratio: 0.0, ctr_coefficient: 0.0

[INFO 03-02 09:32:52] ax.api.client: Trial 8 marked COMPLETED.

Total clicks: 13, impressions: 48436, budget spent: 569323.0

[INFO 03-02 09:32:54] ax.api.client: Generated new trial 9 with parameters {'base_bid': 21.180327, 'bid_to_pay_ratio': 0.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130610 with base_bid: 21.180326915984416, bid_to_pay_ratio: 0.0, ctr_coefficient: 0.0

[INFO 03-02 09:33:40] ax.api.client: Trial 9 marked COMPLETED.

Total clicks: 28, impressions: 70486, budget spent: 1021558.0

[INFO 03-02 09:33:41] ax.api.client: Generated new trial 10 with parameters {'base_bid': 1.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 1.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130611 with base_bid: 1.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 1.0

[INFO 03-02 09:34:28] ax.api.client: Trial 10 marked COMPLETED.

Total clicks: 14, impressions: 3667, budget spent: 49781.0

[INFO 03-02 09:34:30] ax.api.client: Generated new trial 11 with parameters {'base_bid': 100.0, 'bid_to_pay_ratio': 0.0, 'ctr_coefficient': 1.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 100.0, bid_to_pay_ratio: 0.0, ctr_coefficient: 1.0

[INFO 03-02 09:35:18] ax.api.client: Trial 11 marked COMPLETED.

Total clicks: 394, impressions: 447531, budget spent: 30300242.0

[INFO 03-02 09:35:19] ax.api.client: Generated new trial 12 with parameters {'base_bid': 10.420863, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 1.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130607 with base_bid: 10.420862989078394, bid_to_pay_ratio: 1.0, ctr_coefficient: 1.0

[INFO 03-02 09:36:15] ax.api.client: Trial 12 marked COMPLETED.

Total clicks: 104, impressions: 193296, budget spent: 6001817.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:36:17] ax.api.client: Generated new trial 13 with parameters {'base_bid': 25.596123, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130608 with base_bid: 25.59612318427428, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0

[INFO 03-02 09:37:02] ax.api.client: Trial 13 marked COMPLETED.

Total clicks: 76, impressions: 60886, budget spent: 1328047.0

[INFO 03-02 09:37:04] ax.api.client: Generated new trial 14 with parameters {'base_bid': 33.008568, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609 with base_bid: 33.0085677499977, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0

[INFO 03-02 09:37:48] ax.api.client: Trial 14 marked COMPLETED.

Total clicks: 78, impressions: 73066, budget spent: 1874753.0

[INFO 03-02 09:37:51] ax.api.client: Generated new trial 15 with parameters {'base_bid': 41.989179, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130610 with base_bid: 41.98917927924867, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0

[INFO 03-02 09:38:37] ax.api.client: Trial 15 marked COMPLETED.

Total clicks: 77, impressions: 110361, budget spent: 3651395.0

[INFO 03-02 09:38:40] ax.api.client: Generated new trial 16 with parameters {'base_bid': 24.727452, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.184092, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130611 with base_bid: 24.727451764094038, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.18409228051179216

[INFO 03-02 09:39:28] ax.api.client: Trial 16 marked COMPLETED.

Total clicks: 101, impressions: 106901, budget spent: 3446002.0

[INFO 03-02 09:39:31] ax.api.client: Generated new trial 17 with parameters {'base_bid': 26.105294, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.228766, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 26.10529404976349, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.2287657295391852

[INFO 03-02 09:40:21] ax.api.client: Trial 17 marked COMPLETED.

Total clicks: 76, impressions: 118703, budget spent: 3153435.0

[INFO 03-02 09:40:24] ax.api.client: Generated new trial 18 with parameters {'base_bid': 45.775866, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130607 with base_bid: 45.775865583918026, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0

[INFO 03-02 09:41:17] ax.api.client: Trial 18 marked COMPLETED.

Total clicks: 104, impressions: 192667, budget spent: 5966350.0

[INFO 03-02 09:41:20] ax.api.client: Generated new trial 19 with parameters {'base_bid': 17.939555, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.011912, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130608 with base_bid: 17.939554979104212, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.011912448896306826

[INFO 03-02 09:42:03] ax.api.client: Trial 19 marked COMPLETED.

Total clicks: 62, impressions: 39513, budget spent: 734695.0

[INFO 03-02 09:42:06] ax.api.client: Generated new trial 20 with parameters {'base_bid': 1.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.542875, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609 with base_bid: 1.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.5428752978380923

[INFO 03-02 09:42:51] ax.api.client: Trial 20 marked COMPLETED.

Total clicks: 2, impressions: 1374, budget spent: 17976.0

[INFO 03-02 09:42:53] ax.api.client: Generated new trial 21 with parameters {'base_bid': 100.0, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130610 with base_bid: 100.0, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0

[INFO 03-02 09:43:40] ax.api.client: Trial 21 marked COMPLETED.

Total clicks: 263, impressions: 284998, budget spent: 14514934.0

[INFO 03-02 09:43:42] ax.api.client: Generated new trial 22 with parameters {'base_bid': 9.423916, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.105275, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130611 with base_bid: 9.423915636395146, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.10527508604260369

[INFO 03-02 09:44:28] ax.api.client: Trial 22 marked COMPLETED.

Total clicks: 38, impressions: 24319, budget spent: 427493.0

[INFO 03-02 09:44:31] ax.api.client: Generated new trial 23 with parameters {'base_bid': 21.884473, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.360856, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 21.884473191944863, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.3608558925326954

[INFO 03-02 09:45:18] ax.api.client: Trial 23 marked COMPLETED.

Total clicks: 83, impressions: 136393, budget spent: 4036077.0

[INFO 03-02 09:45:22] ax.api.client: Generated new trial 24 with parameters {'base_bid': 37.753477, 'bid_to_pay_ratio': 0.388859, 'ctr_coefficient': 0.0, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130607 with base_bid: 37.753477373175535, bid_to_pay_ratio: 0.38885857944470004, ctr_coefficient: 0.0

[INFO 03-02 09:46:12] ax.api.client: Trial 24 marked COMPLETED.

Total clicks: 71, impressions: 161223, budget spent: 3429396.0

[INFO 03-02 09:46:15] ax.api.client: Generated new trial 25 with parameters {'base_bid': 28.315128, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.293772, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130608 with base_bid: 28.315128375469648, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.2937717585160349

[INFO 03-02 09:46:59] ax.api.client: Trial 25 marked COMPLETED.

Total clicks: 134, impressions: 150665, budget spent: 5270863.0

[INFO 03-02 09:47:02] ax.api.client: Generated new trial 26 with parameters {'base_bid': 28.706611, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.241651, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130609 with base_bid: 28.706611009561584, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.24165097949465517

[INFO 03-02 09:47:46] ax.api.client: Trial 26 marked COMPLETED.

Total clicks: 118, impressions: 141699, budget spent: 4671988.0

[INFO 03-02 09:47:50] ax.api.client: Generated new trial 27 with parameters {'base_bid': 19.354264, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.0, 'pacing_coefficient': 0.0} using GenerationNode MBM.

Simulating date: 20130610 with base_bid: 19.35426426818519, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.0

[INFO 03-02 09:48:35] ax.api.client: Trial 27 marked COMPLETED.

Total clicks: 37, impressions: 40830, budget spent: 850950.0

[INFO 03-02 09:48:38] ax.api.client: Generated new trial 28 with parameters {'base_bid': 28.393223, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.072518, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130611 with base_bid: 28.393223041070982, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.07251784949060883

[INFO 03-02 09:49:24] ax.api.client: Trial 28 marked COMPLETED.

Total clicks: 89, impressions: 75579, budget spent: 1893359.0

[INFO 03-02 09:49:27] ax.api.client: Generated new trial 29 with parameters {'base_bid': 31.951457, 'bid_to_pay_ratio': 1.0, 'ctr_coefficient': 0.276602, 'pacing_coefficient': 1.0} using GenerationNode MBM.

Simulating date: 20130612 with base_bid: 31.95145702462825, bid_to_pay_ratio: 1.0, ctr_coefficient: 0.2766016762347883

[INFO 03-02 09:50:14] ax.api.client: Trial 29 marked COMPLETED.

Total clicks: 98, impressions: 156732, budget spent: 5011977.0

frontier = client.get_pareto_frontier()

data = []
for parameters, metrics, trial_index, arm_name in frontier:
    data.append(
        {
            "budget_spent": metrics["budget_spent"][0],
            "clicks": metrics["clicks"][0],
            "trial_index": trial_index,
            "base_bid": parameters["base_bid"],
            "bid_to_pay_ratio": parameters["bid_to_pay_ratio"],
            "ctr_coefficient": parameters["ctr_coefficient"],
        }
    )

df_pareto = pd.DataFrame(data)
df_pareto = df_pareto[df_pareto["budget_spent"] > 0]

display(df_pareto.sort_values(by="budget_spent", ascending=True))

plt.figure(figsize=(8, 6))
plt.scatter(
    df_pareto["budget_spent"],
    df_pareto["clicks"],
    c="blue",
    edgecolor="k",
    alpha=0.7,
)
plt.xlabel("budget_spent")
plt.ylabel("clicks")
plt.title("Pareto Frontier: budget_spent vs clicks")
plt.grid(True)
plt.show()

Here we can see only a few set of hyperparameters were found in the Pareto Frontier.

The limitation with the optimization of clicks is that the number of clicks is usually low, so to get a good number of them we need to run each trial for a long time.

2.4.6.2 Maximize the number of impressions while minimizing the budget spent¶

This time, minimizing the budget spent and maximizing the number of impressions.

# Now the budget should be set to a number that doesn't limit the bids. We still have to provide one.
budget = 100000000000.0
trial_size = 100000
# In this case the pacing_coefficient should be 0 because the budget is already too large.
# Learning from the previous experiment, we don't care about the predicted CTR, so the coefficient is set to 0.

client = Client()

parameters = [
    RangeParameterConfig(
        name="base_bid", parameter_type="float", bounds=(1, 100)
    ),
    RangeParameterConfig(
        name="bid_to_pay_ratio", parameter_type="float", bounds=(0, 1)
    ),
]

client.configure_experiment(parameters=parameters)
client.configure_optimization(
    objective="-budget_spent, impressions",
    outcome_constraints=["budget_spent <= 6009888.0", "impressions >= 1"],
)

for date in [
    "20130607",
    "20130608",
    "20130609",
    "20130610",
    "20130611",
    "20130612",
]:
    offset = 0
    while True:
        trials = client.get_next_trials(max_trials=1)
        trial_index = list(trials.keys())[0]
        parameters = trials[trial_index]

        base_bid = parameters["base_bid"]
        bid_to_pay_ratio = parameters["bid_to_pay_ratio"]

        print(
            f"Simulating date: {date}, offset: {offset} with base_bid: {base_bid}, bid_to_pay_ratio: {bid_to_pay_ratio}"
        )
        (
            total_bids,
            total_bids_done,
            total_bids_won,
            total_budget_spent,
            total_clicks,
        ) = dsp_simulation(
            date,
            base_bid,
            budget,
            bid_to_pay_ratio,
            0,
            0,
            offset=offset,
            limit=trial_size,
        )
        offset += trial_size
        if total_bids <= 0:
            break
        print(
            f"Total clicks: {total_clicks}, impressions: {total_bids_won}, budget spent: {total_budget_spent}"
        )

        client.complete_trial(
            trial_index=trial_index,
            raw_data={
                "clicks": total_clicks,
                "impressions": total_bids_won,
                "budget_spent": total_budget_spent,
            },
        )

[INFO 03-02 09:50:16] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(node_name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, model_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(node_name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, model_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 03-02 09:50:16] ax.api.client: Generated new trial 0 with parameters {'base_bid': 50.5, 'bid_to_pay_ratio': 0.5} using GenerationNode CenterOfSearchSpace.

Simulating date: 20130607, offset: 0 with base_bid: 50.5, bid_to_pay_ratio: 0.5

[INFO 03-02 09:50:27] ax.api.client: Trial 0 marked COMPLETED.
[INFO 03-02 09:50:27] ax.api.client: Generated new trial 1 with parameters {'base_bid': 14.525911, 'bid_to_pay_ratio': 0.706349} using GenerationNode Sobol.

Total clicks: 21, impressions: 44839, budget spent: 1284016.0
Simulating date: 20130607, offset: 100000 with base_bid: 14.525911077857018, bid_to_pay_ratio: 0.7063486576080322

[INFO 03-02 09:50:38] ax.api.client: Trial 1 marked COMPLETED.
[INFO 03-02 09:50:38] ax.api.client: Generated new trial 2 with parameters {'base_bid': 84.50781, 'bid_to_pay_ratio': 0.225202} using GenerationNode Sobol.

Total clicks: 5, impressions: 6423, budget spent: 86265.0
Simulating date: 20130607, offset: 200000 with base_bid: 84.5078097563237, bid_to_pay_ratio: 0.22520187497138977

[INFO 03-02 09:50:49] ax.api.client: Trial 2 marked COMPLETED.
[INFO 03-02 09:50:49] ax.api.client: Generated new trial 3 with parameters {'base_bid': 59.382288, 'bid_to_pay_ratio': 0.802014} using GenerationNode Sobol.

Total clicks: 45, impressions: 80257, budget spent: 3309767.0
Simulating date: 20130607, offset: 300000 with base_bid: 59.38228816445917, bid_to_pay_ratio: 0.8020140491425991

[INFO 03-02 09:51:00] ax.api.client: Trial 3 marked COMPLETED.
[INFO 03-02 09:51:00] ax.api.client: Generated new trial 4 with parameters {'base_bid': 39.669607, 'bid_to_pay_ratio': 0.25057} using GenerationNode Sobol.

Total clicks: 36, impressions: 58174, budget spent: 2017696.0
Simulating date: 20130607, offset: 400000 with base_bid: 39.669606949202716, bid_to_pay_ratio: 0.2505699433386326

[INFO 03-02 09:51:08] ax.api.client: Trial 4 marked COMPLETED.

Total clicks: 10, impressions: 30129, budget spent: 630521.0

[INFO 03-02 09:51:09] ax.api.client: Generated new trial 5 with parameters {'base_bid': 100.0, 'bid_to_pay_ratio': 1.0} using GenerationNode MBM.

Simulating date: 20130607, offset: 500000 with base_bid: 100.0, bid_to_pay_ratio: 1.0

[INFO 03-02 09:51:11] ax.api.client: Generated new trial 6 with parameters {'base_bid': 70.24953, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130608, offset: 0 with base_bid: 70.24952962176852, bid_to_pay_ratio: 0.0

[INFO 03-02 09:51:21] ax.api.client: Trial 6 marked COMPLETED.

Total clicks: 42, impressions: 59011, budget spent: 2393559.0

[INFO 03-02 09:51:23] ax.api.client: Generated new trial 7 with parameters {'base_bid': 100.0, 'bid_to_pay_ratio': 0.467183} using GenerationNode MBM.

Simulating date: 20130608, offset: 100000 with base_bid: 100.0, bid_to_pay_ratio: 0.4671827525273347

[INFO 03-02 09:51:33] ax.api.client: Trial 7 marked COMPLETED.

Total clicks: 58, impressions: 76367, budget spent: 4001426.0

[INFO 03-02 09:51:35] ax.api.client: Generated new trial 8 with parameters {'base_bid': 1.0, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130608, offset: 200000 with base_bid: 1.0, bid_to_pay_ratio: 0.0

[INFO 03-02 09:51:45] ax.api.client: Trial 8 marked COMPLETED.

Total clicks: 0, impressions: 0, budget spent: 0.0

[INFO 03-02 09:51:46] ax.api.client: Generated new trial 9 with parameters {'base_bid': 31.5848, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130608, offset: 300000 with base_bid: 31.58479959243959, bid_to_pay_ratio: 0.0

[INFO 03-02 09:51:57] ax.api.client: Trial 9 marked COMPLETED.

Total clicks: 10, impressions: 25232, budget spent: 427020.0

[INFO 03-02 09:51:59] ax.api.client: Generated new trial 10 with parameters {'base_bid': 73.075751, 'bid_to_pay_ratio': 1.0} using GenerationNode MBM.

Simulating date: 20130608, offset: 400000 with base_bid: 73.07575134349607, bid_to_pay_ratio: 1.0

[INFO 03-02 09:52:01] ax.api.client: Trial 10 marked COMPLETED.

Total clicks: 10, impressions: 7435, budget spent: 313416.0

[INFO 03-02 09:52:03] ax.api.client: Generated new trial 11 with parameters {'base_bid': 48.766401, 'bid_to_pay_ratio': 1.0} using GenerationNode MBM.

Simulating date: 20130608, offset: 500000 with base_bid: 48.76640103866652, bid_to_pay_ratio: 1.0

[INFO 03-02 09:52:04] ax.api.client: Generated new trial 12 with parameters {'base_bid': 95.447576, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130609, offset: 0 with base_bid: 95.4475756518465, bid_to_pay_ratio: 0.0

[INFO 03-02 09:52:15] ax.api.client: Trial 12 marked COMPLETED.

Total clicks: 51, impressions: 81599, budget spent: 4018088.0

[INFO 03-02 09:52:17] ax.api.client: Generated new trial 13 with parameters {'base_bid': 92.179228, 'bid_to_pay_ratio': 0.196985} using GenerationNode MBM.

Simulating date: 20130609, offset: 100000 with base_bid: 92.17922752789664, bid_to_pay_ratio: 0.1969854975692507

[INFO 03-02 09:52:28] ax.api.client: Trial 13 marked COMPLETED.

Total clicks: 60, impressions: 80659, budget spent: 4240424.0

[INFO 03-02 09:52:30] ax.api.client: Generated new trial 14 with parameters {'base_bid': 37.288643, 'bid_to_pay_ratio': 1.0} using GenerationNode MBM.

Simulating date: 20130609, offset: 200000 with base_bid: 37.28864260266437, bid_to_pay_ratio: 1.0

[INFO 03-02 09:52:42] ax.api.client: Trial 14 marked COMPLETED.

Total clicks: 15, impressions: 20819, budget spent: 552535.0

[INFO 03-02 09:52:43] ax.api.client: Generated new trial 15 with parameters {'base_bid': 74.848518, 'bid_to_pay_ratio': 0.616242} using GenerationNode MBM.

Simulating date: 20130609, offset: 300000 with base_bid: 74.84851819570636, bid_to_pay_ratio: 0.6162422422224937

[INFO 03-02 09:52:54] ax.api.client: Trial 15 marked COMPLETED.

Total clicks: 62, impressions: 56635, budget spent: 2419266.0

[INFO 03-02 09:52:55] ax.api.client: Generated new trial 16 with parameters {'base_bid': 68.840592, 'bid_to_pay_ratio': 0.3719} using GenerationNode MBM.

Simulating date: 20130609, offset: 400000 with base_bid: 68.8405919311658, bid_to_pay_ratio: 0.37189959983295856

[INFO 03-02 09:52:58] ax.api.client: Trial 16 marked COMPLETED.

Total clicks: 13, impressions: 11089, budget spent: 434920.0

[INFO 03-02 09:53:00] ax.api.client: Generated new trial 17 with parameters {'base_bid': 58.15546, 'bid_to_pay_ratio': 0.207723} using GenerationNode MBM.

Simulating date: 20130609, offset: 500000 with base_bid: 58.155459950735604, bid_to_pay_ratio: 0.20772293838213646

[INFO 03-02 09:53:02] ax.api.client: Generated new trial 18 with parameters {'base_bid': 5.637121, 'bid_to_pay_ratio': 0.44961} using GenerationNode MBM.

Simulating date: 20130610, offset: 0 with base_bid: 5.6371205907058135, bid_to_pay_ratio: 0.44961033988468807

[INFO 03-02 09:53:13] ax.api.client: Trial 18 marked COMPLETED.

Total clicks: 1, impressions: 981, budget spent: 9084.0

[INFO 03-02 09:53:15] ax.api.client: Generated new trial 19 with parameters {'base_bid': 74.281912, 'bid_to_pay_ratio': 0.266702} using GenerationNode MBM.

Simulating date: 20130610, offset: 100000 with base_bid: 74.28191177059877, bid_to_pay_ratio: 0.26670193925638164

[INFO 03-02 09:53:26] ax.api.client: Trial 19 marked COMPLETED.

Total clicks: 48, impressions: 60930, budget spent: 2699934.0

[INFO 03-02 09:53:28] ax.api.client: Generated new trial 20 with parameters {'base_bid': 100.0, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 200000 with base_bid: 100.0, bid_to_pay_ratio: 0.0

[INFO 03-02 09:53:39] ax.api.client: Trial 20 marked COMPLETED.

Total clicks: 70, impressions: 82731, budget spent: 4220104.0

[INFO 03-02 09:53:40] ax.api.client: Generated new trial 21 with parameters {'base_bid': 48.00088, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 300000 with base_bid: 48.00087989369378, bid_to_pay_ratio: 0.0

[INFO 03-02 09:53:51] ax.api.client: Trial 21 marked COMPLETED.

Total clicks: 19, impressions: 31950, budget spent: 762060.0

[INFO 03-02 09:53:53] ax.api.client: Generated new trial 22 with parameters {'base_bid': 1.0, 'bid_to_pay_ratio': 1.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 400000 with base_bid: 1.0, bid_to_pay_ratio: 1.0

[INFO 03-02 09:53:57] ax.api.client: Trial 22 marked COMPLETED.

Total clicks: 0, impressions: 4, budget spent: 38.0

[INFO 03-02 09:53:59] ax.api.client: Generated new trial 23 with parameters {'base_bid': 77.849288, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130610, offset: 500000 with base_bid: 77.84928807524392, bid_to_pay_ratio: 0.0

[INFO 03-02 09:54:01] ax.api.client: Generated new trial 24 with parameters {'base_bid': 62.236705, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 0 with base_bid: 62.236705184003796, bid_to_pay_ratio: 0.0

[INFO 03-02 09:54:12] ax.api.client: Trial 24 marked COMPLETED.

Total clicks: 26, impressions: 48951, budget spent: 1676293.0

[INFO 03-02 09:54:14] ax.api.client: Generated new trial 25 with parameters {'base_bid': 20.150996, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 100000 with base_bid: 20.150996043956805, bid_to_pay_ratio: 0.0

[INFO 03-02 09:54:25] ax.api.client: Trial 25 marked COMPLETED.

Total clicks: 2, impressions: 11797, budget spent: 156515.0

[INFO 03-02 09:54:27] ax.api.client: Generated new trial 26 with parameters {'base_bid': 86.044212, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 200000 with base_bid: 86.0442120817592, bid_to_pay_ratio: 0.0

[INFO 03-02 09:54:38] ax.api.client: Trial 26 marked COMPLETED.

Total clicks: 66, impressions: 80108, budget spent: 3935892.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/botorch/optim/optimize.py:789: RuntimeWarning:

Optimization failed in `gen_candidates_scipy` with the following warning(s):
[NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal')]
Trying again with a new set of initial conditions.

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:54:42] ax.api.client: Generated new trial 27 with parameters {'base_bid': 53.670708, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 300000 with base_bid: 53.670708420373714, bid_to_pay_ratio: 0.0

[INFO 03-02 09:54:53] ax.api.client: Trial 27 marked COMPLETED.

Total clicks: 28, impressions: 42228, budget spent: 1354027.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/botorch/optim/optimize.py:789: RuntimeWarning:

Optimization failed in `gen_candidates_scipy` with the following warning(s):
[NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal')]
Trying again with a new set of initial conditions.

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/botorch/optim/optimize.py:789: RuntimeWarning:

Optimization failed on the second try, after generating a new set of initial conditions.

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:54:58] ax.api.client: Generated new trial 28 with parameters {'base_bid': 40.576232, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130611, offset: 400000 with base_bid: 40.576232109517726, bid_to_pay_ratio: 0.0

[INFO 03-02 09:55:03] ax.api.client: Trial 28 marked COMPLETED.

Total clicks: 12, impressions: 11894, budget spent: 240019.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:55:05] ax.api.client: Generated new trial 29 with parameters {'base_bid': 100.0, 'bid_to_pay_ratio': 0.231523} using GenerationNode MBM.

Simulating date: 20130611, offset: 500000 with base_bid: 100.0, bid_to_pay_ratio: 0.2315226432077734

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/botorch/optim/optimize.py:789: RuntimeWarning:

Optimization failed in `gen_candidates_scipy` with the following warning(s):
[NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL: .'), NumericalWarning('A not p.d., added jitter of 1.0e-08 to the diagonal')]
Trying again with a new set of initial conditions.

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/botorch/optim/optimize.py:789: RuntimeWarning:

Optimization failed on the second try, after generating a new set of initial conditions.

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:55:12] ax.api.client: Generated new trial 30 with parameters {'base_bid': 74.503584, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 0 with base_bid: 74.5035836418182, bid_to_pay_ratio: 0.0

[INFO 03-02 09:55:23] ax.api.client: Trial 30 marked COMPLETED.

Total clicks: 55, impressions: 75657, budget spent: 3408020.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:55:25] ax.api.client: Generated new trial 31 with parameters {'base_bid': 66.294621, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 100000 with base_bid: 66.29462145392186, bid_to_pay_ratio: 0.0

[INFO 03-02 09:55:36] ax.api.client: Trial 31 marked COMPLETED.

Total clicks: 27, impressions: 51874, budget spent: 1883881.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:55:38] ax.api.client: Generated new trial 32 with parameters {'base_bid': 58.931006, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 200000 with base_bid: 58.93100588551603, bid_to_pay_ratio: 0.0

[INFO 03-02 09:55:49] ax.api.client: Trial 32 marked COMPLETED.

Total clicks: 25, impressions: 46030, budget spent: 1412536.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:55:52] ax.api.client: Generated new trial 33 with parameters {'base_bid': 10.442717, 'bid_to_pay_ratio': 0.848654} using GenerationNode MBM.

Simulating date: 20130612, offset: 300000 with base_bid: 10.442716848315674, bid_to_pay_ratio: 0.8486540035281105

[INFO 03-02 09:56:03] ax.api.client: Trial 33 marked COMPLETED.

Total clicks: 1, impressions: 3901, budget spent: 44696.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:56:05] ax.api.client: Generated new trial 34 with parameters {'base_bid': 81.808256, 'bid_to_pay_ratio': 0.0} using GenerationNode MBM.

Simulating date: 20130612, offset: 400000 with base_bid: 81.80825599902185, bid_to_pay_ratio: 0.0

[INFO 03-02 09:56:11] ax.api.client: Trial 34 marked COMPLETED.

Total clicks: 26, impressions: 38884, budget spent: 1798288.0

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:56:14] ax.api.client: Generated new trial 35 with parameters {'base_bid': 30.272029, 'bid_to_pay_ratio': 0.393018} using GenerationNode MBM.

Simulating date: 20130612, offset: 500000 with base_bid: 30.27202936626861, bid_to_pay_ratio: 0.3930175692019658

frontier = client.get_pareto_frontier()

data = []
for parameters, metrics, trial_index, arm_name in frontier:
    data.append(
        {
            "budget_spent": metrics["budget_spent"][0],
            "impressions": metrics["impressions"][0],
            "trial_index": trial_index,
            "base_bid": parameters["base_bid"],
            "bid_to_pay_ratio": parameters["bid_to_pay_ratio"],
        }
    )

df_pareto = pd.DataFrame(data)
df_pareto = df_pareto[df_pareto["budget_spent"] > 0]

display(df_pareto.sort_values(by="budget_spent", ascending=True))

plt.figure(figsize=(8, 6))
plt.scatter(
    df_pareto["budget_spent"],
    df_pareto["impressions"],
    c="blue",
    edgecolor="k",
    alpha=0.7,
)
plt.xlabel("budget_spent")
plt.ylabel("impressions")
plt.title("Pareto Frontier: budget_spent vs clicks")
plt.grid(True)
plt.show()

For impressions the optimization algorithm was able to find more optimal results. This is because the higher number of impressions gives more information to the Bayesian Optimization about the relationship between input and outcomes.

3. Bayesian Optimization on Multi-Armed Bandits¶

A Multi-Armed Bandit is a problem where there are multiple arms (actions) to choose from. A/B Testing is a special case of them.

3.1. A/B Testing¶

In A/B Testing, we have a set of arms $A_1, A_2, ..., A_n$ and we want to find the best arm.

The arms are the different versions of the product or feature that we want to test. The outcome is the success of the arm.

The goal is to find the best arm to maximize the success.

The problem is that we don’t know the success of the arms before running the test.

In A/B Testing the criteria to present an arm to the user is randomized to eliminate potential confounders, similar to an RCT.

3.2. Policy Exploration Problem¶

Through exploration, we want to find the best policy to maximize the outcome. The policy is defined by a “scoring function”.

This means, we have a set of features about the user and the content, then a prediction model will return a series of predictions from those features (E.g.: click probability, share probability, etc). We want to define how those predictions will be weighted to select the arm to maximize the outcome. Examples of this are:

Facebook feed: Decide which content to show to the user.
LinkedIn search results: Who should appear first in the search results.
Netflix recommendations: Which movies to recommend to the user.

We define the features corresponding to the User (Who is receiving the content) and the ones corresponding to the Content (What is being shown to the user) as $u$ and $c$ .

A Prediction Model generates $d$ predictions $f_i(u, c)$ (E.g.: click probability, share probability, etc.) based on the user and content.

We have to weigh in those predictions to calculate the score $s(u, c) = \sum_{i=1}^d x_i f_i(u, c)$ . The score will decide which content is shown to the user.

The policy is the definition of $x_i$ which are the weights assigned to each prediction.

3.3. Sequential Optimization¶

A/B Testing models can be used to optimize the policy. This is what the Meta Adaptive Experimentation Team currently does.

Bayesian Optimization can be used to test different policies in a sequential manner.

Bayesian Optimization does exploration and exploitation, this means, it will try to use the policies that are returning the best results but also try new policies to explore the input space.

3.4. Literature¶

The following literature shows how Bayesian Optimization techniques have been applied by top companies to optimize their advertising algorithms.

Google Vizier: A Service for Black-Box Optimization (2017): Google. Comparison of Bayesian Optimization vs Simulated Annealing. Initial definitions of Bayesian Optimization.
Online Parameter Selection for Web-based Ranking Problems (2018): LinkedIn. Select the policy to score and rank search results.
Constrained Bayesian Optimization with Noisy Experiments (2018): Meta. Improvements to Bayesian Optimization. Quasi-Monte Carlo sampling.
Bayesian Optimization for Policy Search via Online-Offline Experimentation (2019): Meta. Mix online (Real users) and offline (Simulations) experiments. Use Multi-task Gaussian Process (MTGP) to combine the results and model the response surface.
Experimenting, Fast and Slow: Bayesian Optimization of Long-term Outcomes with Online Experiments (2025): Combine short-run experiments (SRE) and long-run experiments (LRE) with MTGP and Target-Aware Gaussian Process Model (TAGP) to model the response surface. Use proxy metrics for short-run experiments.

3.5. Multi-Armed Bandit Problem¶

We have $n$ Arms -> $A_1, A_2, ..., A_n$ , and we need to find the best arm.

Maximizing $Reward(A_i)$ requires testing, but the number of tests is limited.

Every time we didn’t pull the best arm, we incur a regret.

Regret: The difference between the best arm and the arm we chose.

In other words, how much we would have won if we used the best arm instead of the one we tried.

3.5.1. Traditional Exploration-Exploitation Algorithms¶

To minimize the regret we need to exploit the arm with highest rewards, but we also need to explore to be sure we are not missing any arm that could give us better results.

There are different techniques to balance exploration and exploitation.

Classic A/B Testing

All bandits are tried equally often. The policy doesn’t adapt to the results. Regret increases linearly $O(T)$

UCB1 (Upper Confidence Bound)

On each iteration, the expected reward of each arm is calculated based on the mean reward and the number of times the arm has been pulled.

If we have an arm $A_i$ , at iteration $t$ , then:

$UCB(A_i) = \hat{\mu}_i + \kappa \sqrt{\frac{2 \log(t)}{N_i(t)}}$

$\kappa$ -> exploration vs exploitation trade-off
$\hat{\mu}_i$ -> the mean reward of arm $A_i$ (Frequentist estimator)
$N_i(t)$ -> the number of times arm $A_i$ has been pulled until time $t$

Without uncertainty (The mean reward is stable) regret is $O(log T)$ . This technique is a frequentist approach, and widely used in the field. It assumes there is a ground truth for the reward of each arm. (E.g., CTR is 0.1 for a given arm and we can estimate it)

Ref: Finite-time Analysis of the Multiarmed Bandit Problem (Peter Auer, 2002)

Ref: Introduction to Multi-Armed Bandits (Aleksandrs Slivkins, 2019)

3.5.2. Bayesian Algorithms¶

Bayesian algorithms consider the uncertainty of the rewards. This means, there isn’t a simple “ground truth”, like a single expected CTR for an arm, but a distribution of possible CTRs.

3.5.2.1. Thompson Sampling¶

This algorithm defines a prior that is updated on each iteration.

Prior: $P(A_i) \sim Beta(\alpha_i, \beta_i)$ | $\alpha_i = 1, \beta_i = 1$

Arm Selection: Evaluate the Beta distribution for each $A_i$ , should return a value for $P(A_i)$ . The arm with the highest value is selected.

Evaluation: Evaluate A_i in a real experiment. If it’s a simulation do $Bernoulli(P(A_i))$ .

Posterior: If $Bernoulli(P(A_i))$ is 1, update $\alpha_i = \alpha_i + 1$ else $\beta_i = \beta_i + 1$

Regret is $O(log T)$ . Thompson Sampling supports higher variance.

Note: Thompson Sampling is a simplified version of the analytical solution for the Probabilistic Programming scenario of the coin toss problem. It’s a Heuristic approach.

Ref: Analysis of Thompson Sampling for the Multi-armed Bandit Problem (Shipra Agrawal and Navin Goyal, 2012)

Ref: A Tutorial on Thompson Sampling (Russo, 2018)

3.5.2.2. Gaussian Process Bandit¶

Gaussian Process Bandit is a Bayesian approach to the Multi-Armed Bandit problem. It uses a Gaussian Process to model the uncertainty of the rewards.

Reminder: GP is defined by $f(x) \sim GP(m(x), k(x, x'))$ where:

$x$ represents the arm
$f(x)$ represents the reward of the arm, is maximized
Regret is minimized

In the Multi-Armed Bandit problem, the input $x_i$ is the probability of choosing arm $A_i$ .

In a classic Multi-Armed Bandit with $n$ options, the Gaussian Process will be linear. One of the solutions is better and the maximum is on the border of the search space. The ideal function would be an hyperplane of the search space.

Compared to Thompson Sampling, a GP Bandit supports the case where there is a non-linear relationship between the chosen arm and the reward.

Example of Gaussian Process Bandit:

Arm 1 has a CTR of 0.1 and Arm 2 has a CTR of 0.2, they are unknown.
The bandit will explore both arms, with a probability for each one $P(A_1)$ and $P(A_2)$ .
The function $f(P(A_1),P(A_2))$ is the reward based on the probability of choosing Arm 1 or Arm 2.
The function is linear, assuming each Arm has a constant CTR.

Ref: Weighted Gaussian Process Bandits for Non-stationary Environments (2021)

3.5.3. Hypothesis¶

We start with a Multi-Armed Bandit problem, and define a hypothesis. Then we’ll run a simulation to test this hypothesis.

Scenario

We have $n$ Arms $A_1, A_2, ..., A_n$
Each arm has an unknown reward $f(A_i)$ (E.g.: Click-through rate)
$T$ : Total number of pulls (Number of times one of the arms is presented to the user)
Experiment: Variation of the frequency of the arms $x_1, x_2, ..., x_n$ (Each arm has a probability of being chosen)
Exploitation: Pull the arm with the estimated best reward
Exploration: Pull the arm with the highest uncertainty
Regret: At the end of the experiment we can get the reward of the best arm, and calculate the regret if we have always pulled the best arm.

Hypothesis

UCB1 is the best algorithm when each Arm has a constant reward.
Reward is linear in theory but not in practice due to variance (E.g.: CTR isn’t really constant).
Thompson Sampling and GP-Bandit are more robust to non-linear relationships with high variance.

Note: Records show that GP-Bandit achieves better results in practice but it’s not guaranteed by the theoretical analysis.

Ref: Weighted Gaussian Process Bandits for Non-stationary Environments (2021)

Ref: Gaussian Process Upper Confidence Bound Achieves Nearly-Optimal Regret in Noise-Free Gaussian Process Bandits (2025)

3.5.1 Simulation with minimal uncertainty¶

We’ll run and compare the performance of the following algorithms:

A/B Testing
UCB1
Thompson Sampling
GP Bandit

The arms being tested are defined by a ground truth, constant CTR of 0.1, 0.2 and 0.3. The uncertainty is minimal, with a standard deviation of 0.01.

Based on the hypothesis, UCB1 should perform better than A/B Testing and Bayesian Optimization methods (Thompson Sampling and GP Bandit).

The code implemented for the simulation can be found in Ax_utils.py. The code is based on the SimulationExperiment abstract class with concrete implementations for each algorithm.

The GP Bandit algorithm makes use of the Ax library.

print("Starting Simulation Experiment")
print("Initialize algorithms")
ab_testing = AB_Testing(n_arms=3)
ucb1 = UCB1(n_arms=3)
thompson_sampling = ThompsonSampling(n_arms=3)
gp_bandit = GP_Bandit(n_arms=3, batch_size=1000)

print("Setup AB Testing Experiment")
ab_testing_experiment = SimulationExperiment(
    ctr_means=[0.1, 0.2, 0.3], ctr_stds=[0.01, 0.01, 0.01], algorithm=ab_testing
)

print("Setup UCB1 Experiment")
ucb1_experiment = SimulationExperiment(
    ctr_means=[0.1, 0.2, 0.3], ctr_stds=[0.01, 0.01, 0.01], algorithm=ucb1
)

print("Setup Thompson Sampling Experiment")
thompson_sampling_experiment = SimulationExperiment(
    ctr_means=[0.1, 0.2, 0.3],
    ctr_stds=[0.01, 0.01, 0.01],
    algorithm=thompson_sampling,
)

print("Setup GP Bandit Experiment")
gp_bandit_experiment = SimulationExperiment(
    ctr_means=[0.1, 0.2, 0.3], ctr_stds=[0.01, 0.01, 0.01], algorithm=gp_bandit
)

[INFO 03-02 09:56:14] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(node_name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, model_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(node_name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, model_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 03-02 09:56:14] ax.api.client: Generated new trial 0 with parameters {'w_0': 0.5, 'w_1': 0.5} using GenerationNode CenterOfSearchSpace.

Starting Simulation Experiment
Initialize algorithms
Setup AB Testing Experiment
Setup UCB1 Experiment
Setup Thompson Sampling Experiment
Setup GP Bandit Experiment

(
    ab_testing_total_reward,
    ab_testing_total_trials,
    ab_testing_accum_rewards_per_trial,
    ab_testing_pulls_per_arm_per_trial,
    ab_testing_expected_rewards_per_arm,
    ab_testing_regret_per_trial,
) = ab_testing_experiment.run_experiment(n_trials=20000)
print(
    f"AB Testing - Reward: {ab_testing_total_reward}, Trials: {ab_testing_total_trials}"
)

AB Testing - Reward: 3988, Trials: 20000

(
    ucb1_total_reward,
    ucb1_total_trials,
    ucb1_accum_rewards_per_trial,
    ucb1_pulls_per_arm_per_trial,
    ucb1_expected_rewards_per_arm,
    ucb1_regret_per_trial,
) = ucb1_experiment.run_experiment(n_trials=20000)
print(f"UCB1 - Reward: {ucb1_total_reward}, Trials: {ucb1_total_trials}")

UCB1 - Reward: 5528, Trials: 20000

(
    thompson_sampling_total_reward,
    thompson_sampling_total_trials,
    thompson_sampling_accum_rewards_per_trial,
    thompson_sampling_pulls_per_arm_per_trial,
    thompson_sampling_expected_rewards_per_arm,
    thompson_sampling_regret_per_trial,
) = thompson_sampling_experiment.run_experiment(n_trials=20000)
print(
    f"Thompson Sampling - Reward: {thompson_sampling_total_reward}, Trials: {thompson_sampling_total_trials}"
)

Thompson Sampling - Reward: 5933, Trials: 20000

(
    gp_bandit_total_reward,
    gp_bandit_total_trials,
    gp_bandit_accum_rewards_per_trial,
    gp_bandit_pulls_per_arm_per_trial,
    gp_bandit_expected_rewards_per_arm,
    gp_bandit_regret_per_trial,
) = gp_bandit_experiment.run_experiment(n_trials=20000)
print(
    f"GP Bandit - Reward: {gp_bandit_total_reward}, Trials: {gp_bandit_total_trials}"
)

[INFO 03-02 09:56:15] ax.api.client: Trial 0 marked COMPLETED.
[INFO 03-02 09:56:15] ax.api.client: Generated new trial 1 with parameters {'w_0': 0.21406, 'w_1': 0.235322} using GenerationNode Sobol.
[INFO 03-02 09:56:15] ax.api.client: Trial 1 marked COMPLETED.
[INFO 03-02 09:56:15] ax.api.client: Generated new trial 2 with parameters {'w_0': 0.618025, 'w_1': 0.308337} using GenerationNode Sobol.
[INFO 03-02 09:56:15] ax.api.client: Trial 2 marked COMPLETED.
[INFO 03-02 09:56:15] ax.api.client: Generated new trial 3 with parameters {'w_0': 0.260387, 'w_1': 0.445824} using GenerationNode Sobol.
[INFO 03-02 09:56:15] ax.api.client: Trial 3 marked COMPLETED.
[INFO 03-02 09:56:15] ax.api.client: Generated new trial 4 with parameters {'w_0': 0.007549, 'w_1': 0.874498} using GenerationNode Sobol.
[INFO 03-02 09:56:16] ax.api.client: Trial 4 marked COMPLETED.
[INFO 03-02 09:56:18] ax.api.client: Generated new trial 5 with parameters {'w_0': 0.111591, 'w_1': 0.005388} using GenerationNode MBM.
[INFO 03-02 09:56:18] ax.api.client: Trial 5 marked COMPLETED.
[INFO 03-02 09:56:20] ax.api.client: Generated new trial 6 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:56:20] ax.api.client: Trial 6 marked COMPLETED.
[INFO 03-02 09:56:22] ax.api.client: Generated new trial 7 with parameters {'w_0': 0.0, 'w_1': 0.155146} using GenerationNode MBM.
[INFO 03-02 09:56:22] ax.api.client: Trial 7 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:56:25] ax.api.client: Generated new trial 8 with parameters {'w_0': 1.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:56:25] ax.api.client: Trial 8 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:56:29] ax.api.client: Generated new trial 9 with parameters {'w_0': 0.0, 'w_1': 0.055877} using GenerationNode MBM.
[INFO 03-02 09:56:29] ax.api.client: Trial 9 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:56:33] ax.api.client: Generated new trial 10 with parameters {'w_0': 0.057206, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:56:33] ax.api.client: Trial 10 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[WARNING 03-02 09:56:44] ax.generation_strategy.generation_node: gen failed with error GenerationStrategy exceeded `MAX_GEN_ATTEMPTS` of 5 while trying to generate a unique parameterization. This indicates that the search space has likely been fully explored, or that the sweep has converged., switching to fallback model with generator_enum Generators.SOBOL
[INFO 03-02 09:56:44] ax.api.client: Generated new trial 11 with parameters {'w_0': 0.319751, 'w_1': 0.207626} using GenerationNode MBM.
[INFO 03-02 09:56:44] ax.api.client: Trial 11 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:56:48] ax.api.client: Generated new trial 12 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:56:48] ax.api.client: Trial 12 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:56:59] ax.api.client: Generated new trial 13 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:56:59] ax.api.client: Trial 13 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:57:01] ax.api.client: Generated new trial 14 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:57:01] ax.api.client: Trial 14 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:57:12] ax.api.client: Generated new trial 15 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:57:12] ax.api.client: Trial 15 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:57:14] ax.api.client: Generated new trial 16 with parameters {'w_0': 0.022539, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:57:14] ax.api.client: Trial 16 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:57:16] ax.api.client: Generated new trial 17 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:57:16] ax.api.client: Trial 17 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:57:21] ax.api.client: Generated new trial 18 with parameters {'w_0': 0.0, 'w_1': 0.10825} using GenerationNode MBM.
[INFO 03-02 09:57:21] ax.api.client: Trial 18 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:57:27] ax.api.client: Generated new trial 19 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 09:57:27] ax.api.client: Trial 19 marked COMPLETED.
/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

/opt/homebrew/Caskroom/miniconda/base/envs/msml610-project/lib/python3.13/site-packages/linear_operator/utils/cholesky.py:40: NumericalWarning:

A not p.d., added jitter of 1.0e-08 to the diagonal

[INFO 03-02 09:57:32] ax.api.client: Generated new trial 20 with parameters {'w_0': 0.0, 'w_1': 0.0} using GenerationNode MBM.

GP Bandit - Reward: 5039, Trials: 20000

3.5.1.1 Compare Results¶

We’ll plot the results of the experiments.

# Plot accumulated clicks and regret side by side
fig, axes = plt.subplots(1, 2, figsize=(16, 5))

# Accumulated clicks
axes[0].plot(ab_testing_accum_rewards_per_trial, label="AB Testing")
axes[0].plot(ucb1_accum_rewards_per_trial, label="UCB1")
axes[0].plot(
    thompson_sampling_accum_rewards_per_trial, label="Thompson Sampling"
)
axes[0].plot(gp_bandit_accum_rewards_per_trial, label="GP Bandit")
axes[0].set_title("Accumulated Clicks")
axes[0].set_xlabel("Trial")
axes[0].set_ylabel("Accumulated clicks")
axes[0].legend()

# Accumulated regret
axes[1].plot(ab_testing_regret_per_trial, label="AB Testing")
axes[1].plot(ucb1_regret_per_trial, label="UCB1")
axes[1].plot(thompson_sampling_regret_per_trial, label="Thompson Sampling")
axes[1].plot(gp_bandit_regret_per_trial, label="GP Bandit")
axes[1].set_title("Accumulated Regret")
axes[1].set_xlabel("Trial")
axes[1].set_ylabel("Regret")
axes[1].legend()

plt.tight_layout()

Now we’ll plot how each arm was pulled for each algorithm.

# Plot ab_testing_pulls_per_arm_per_trial
fig, axes = plt.subplots(2, 2, figsize=(16, 5))

# AB Testing
for i in range(3):
    axes[0][0].plot(
        ab_testing_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}"
    )
axes[0][0].set_title("AB Testing - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

# UCB1
for i in range(3):
    axes[0][1].plot(ucb1_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}")
axes[0][1].set_title("UCB1 - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

# Thompson Sampling
for i in range(3):
    axes[1][0].plot(
        thompson_sampling_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}"
    )
axes[1][0].set_title("Thompson Sampling - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

# GP Bandit
for i in range(3):
    axes[1][1].plot(
        gp_bandit_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}"
    )
axes[1][1].set_title("GP Bandit - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

plt.tight_layout()

3.5.1.2 Analysis of the results¶

Under a certain and constant CTR with minimal uncertainty, Bayes Optimization (GP Bandit) doesn’t perform well. Statistically it learns which arm is the best, but it keeps exploring other arms.

UCB1 performs better because it’s more focused on exploitation, and assumes that the expected reward is constant, which is the case.

Thompson Sampling surprisingly performs better than UCB1. My hypothesis was that it would have a similar performance or slightly worse, as it doesn’t pick the best arm always, it just runs a probability to do it.

3.5.2 Simulation with higher uncertainty¶

Now we’re going to simulate a different scenario. It tooks real CTRs from the Upworthy Research Archive dataset, where the CTRs are much more close.

The Upworthy Research Archive dataset has been recorded by a viral news website which did A/B testing for news headlines and tried to identify which headlines had a higher click ratio. Here we took one of the articles as an example.

Headline 1: 0.049
Headline 2: 0.040
Headline 3: 0.035

Now, the dataset only contains the final CTR, not the uncertainty. We’ll assume the CTR is not constant, it has a variance, and the probability of the CTR for each headline may even overlap.

Here we’re assuming CTRs distributions are constant over time. In reality they are not, at a different time of the day some headlines could gain more attention as they audience changes.

Std will be 0.01 for all headlines.

# Visualize CTR priors as normal distributions
ctr_means = [0.049, 0.040, 0.035]
ctr_stds = [0.01, 0.01, 0.01]
x = np.linspace(0.0, 0.1, 400)

plt.figure(figsize=(5, 3))
i = 0
for m, s in zip(ctr_means, ctr_stds):
    # Calculate the probability density function for the normal distribution
    pdf = (1 / (s * np.sqrt(2 * np.pi))) * np.exp(-0.5 * ((x - m) / s) ** 2)
    plt.plot(x, pdf, label=f"Arm {i}")
    i += 1

plt.title("CTR Normal Distributions")
plt.legend()
plt.tight_layout()

print("Starting Simulation Experiment")
print("Initialize algorithms")
ab_testing = AB_Testing(n_arms=3)
ucb1 = UCB1(n_arms=3)
thompson_sampling = ThompsonSampling(n_arms=3)
gp_bandit = GP_Bandit(n_arms=3, batch_size=1000)

print("Setup AB Testing Experiment")
ab_testing_experiment = SimulationExperiment(
    ctr_means=ctr_means, ctr_stds=ctr_stds, algorithm=ab_testing
)

print("Setup UCB1 Experiment")
ucb1_experiment = SimulationExperiment(
    ctr_means=ctr_means, ctr_stds=ctr_stds, algorithm=ucb1
)

print("Setup Thompson Sampling Experiment")
thompson_sampling_experiment = SimulationExperiment(
    ctr_means=ctr_means, ctr_stds=ctr_stds, algorithm=thompson_sampling
)

print("Setup GP Bandit Experiment")
gp_bandit_experiment = SimulationExperiment(
    ctr_means=ctr_means, ctr_stds=ctr_stds, algorithm=gp_bandit
)

[INFO 03-02 10:03:30] ax.api.client: GenerationStrategy(name='Center+Sobol+MBM:fast', nodes=[CenterGenerationNode(next_node_name='Sobol'), GenerationNode(node_name='Sobol', generator_specs=[GeneratorSpec(generator_enum=Sobol, model_key_override=None)], transition_criteria=[MinTrials(transition_to='MBM'), MinTrials(transition_to='MBM')]), GenerationNode(node_name='MBM', generator_specs=[GeneratorSpec(generator_enum=BoTorch, model_key_override=None)], transition_criteria=[])]) chosen based on user input and problem structure.
[INFO 03-02 10:03:30] ax.api.client: Generated new trial 0 with parameters {'w_0': 0.5, 'w_1': 0.5} using GenerationNode CenterOfSearchSpace.

Starting Simulation Experiment
Initialize algorithms
Setup AB Testing Experiment
Setup UCB1 Experiment
Setup Thompson Sampling Experiment
Setup GP Bandit Experiment

(
    ab_testing_total_reward,
    ab_testing_total_trials,
    ab_testing_accum_rewards_per_trial,
    ab_testing_pulls_per_arm_per_trial,
    ab_testing_expected_rewards_per_arm,
    ab_testing_regret_per_trial,
) = ab_testing_experiment.run_experiment(n_trials=20000)
print(
    f"AB Testing - Reward: {ab_testing_total_reward}, Trials: {ab_testing_total_trials}"
)

AB Testing - Reward: 769, Trials: 20000

(
    ucb1_total_reward,
    ucb1_total_trials,
    ucb1_accum_rewards_per_trial,
    ucb1_pulls_per_arm_per_trial,
    ucb1_expected_rewards_per_arm,
    ucb1_regret_per_trial,
) = ucb1_experiment.run_experiment(n_trials=20000)
print(f"UCB1 - Reward: {ucb1_total_reward}, Trials: {ucb1_total_trials}")

UCB1 - Reward: 822, Trials: 20000

(
    thompson_sampling_total_reward,
    thompson_sampling_total_trials,
    thompson_sampling_accum_rewards_per_trial,
    thompson_sampling_pulls_per_arm_per_trial,
    thompson_sampling_expected_rewards_per_arm,
    thompson_sampling_regret_per_trial,
) = thompson_sampling_experiment.run_experiment(n_trials=20000)
print(
    f"Thompson Sampling - Reward: {thompson_sampling_total_reward}, Trials: {thompson_sampling_total_trials}"
)

Thompson Sampling - Reward: 1002, Trials: 20000

(
    gp_bandit_total_reward,
    gp_bandit_total_trials,
    gp_bandit_accum_rewards_per_trial,
    gp_bandit_pulls_per_arm_per_trial,
    gp_bandit_expected_rewards_per_arm,
    gp_bandit_regret_per_trial,
) = gp_bandit_experiment.run_experiment(n_trials=20000)
print(
    f"GP Bandit - Reward: {gp_bandit_total_reward}, Trials: {gp_bandit_total_trials}"
)

[INFO 03-02 10:03:38] ax.api.client: Trial 0 marked COMPLETED.
[INFO 03-02 10:03:38] ax.api.client: Generated new trial 1 with parameters {'w_0': 0.090892, 'w_1': 0.410915} using GenerationNode Sobol.
[INFO 03-02 10:03:38] ax.api.client: Trial 1 marked COMPLETED.
[INFO 03-02 10:03:38] ax.api.client: Generated new trial 2 with parameters {'w_0': 0.335635, 'w_1': 0.546278} using GenerationNode Sobol.
[INFO 03-02 10:03:38] ax.api.client: Trial 2 marked COMPLETED.
[INFO 03-02 10:03:38] ax.api.client: Generated new trial 3 with parameters {'w_0': 0.706788, 'w_1': 0.21163} using GenerationNode Sobol.
[INFO 03-02 10:03:38] ax.api.client: Trial 3 marked COMPLETED.
[INFO 03-02 10:03:38] ax.api.client: Generated new trial 4 with parameters {'w_0': 0.474064, 'w_1': 0.094565} using GenerationNode Sobol.
[INFO 03-02 10:03:38] ax.api.client: Trial 4 marked COMPLETED.
[INFO 03-02 10:03:41] ax.api.client: Generated new trial 5 with parameters {'w_0': 0.29486, 'w_1': 0.70514} using GenerationNode MBM.
[INFO 03-02 10:03:41] ax.api.client: Trial 5 marked COMPLETED.
[INFO 03-02 10:03:43] ax.api.client: Generated new trial 6 with parameters {'w_0': 0.263212, 'w_1': 0.297707} using GenerationNode MBM.
[INFO 03-02 10:03:43] ax.api.client: Trial 6 marked COMPLETED.
[INFO 03-02 10:03:45] ax.api.client: Generated new trial 7 with parameters {'w_0': 0.257918, 'w_1': 0.742082} using GenerationNode MBM.
[INFO 03-02 10:03:45] ax.api.client: Trial 7 marked COMPLETED.
[INFO 03-02 10:03:48] ax.api.client: Generated new trial 8 with parameters {'w_0': 0.303202, 'w_1': 0.237899} using GenerationNode MBM.
[INFO 03-02 10:03:48] ax.api.client: Trial 8 marked COMPLETED.
[INFO 03-02 10:03:50] ax.api.client: Generated new trial 9 with parameters {'w_0': 0.307619, 'w_1': 0.692381} using GenerationNode MBM.
[INFO 03-02 10:03:50] ax.api.client: Trial 9 marked COMPLETED.
[INFO 03-02 10:03:53] ax.api.client: Generated new trial 10 with parameters {'w_0': 0.33727, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 10:03:53] ax.api.client: Trial 10 marked COMPLETED.
[INFO 03-02 10:03:56] ax.api.client: Generated new trial 11 with parameters {'w_0': 0.283742, 'w_1': 0.708286} using GenerationNode MBM.
[INFO 03-02 10:03:56] ax.api.client: Trial 11 marked COMPLETED.
[INFO 03-02 10:03:58] ax.api.client: Generated new trial 12 with parameters {'w_0': 0.382327, 'w_1': 0.472681} using GenerationNode MBM.
[INFO 03-02 10:03:58] ax.api.client: Trial 12 marked COMPLETED.
[INFO 03-02 10:04:01] ax.api.client: Generated new trial 13 with parameters {'w_0': 0.888925, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 10:04:01] ax.api.client: Trial 13 marked COMPLETED.
[INFO 03-02 10:04:04] ax.api.client: Generated new trial 14 with parameters {'w_0': 1.0, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 10:04:04] ax.api.client: Trial 14 marked COMPLETED.
[INFO 03-02 10:04:06] ax.api.client: Generated new trial 15 with parameters {'w_0': 0.843178, 'w_1': 0.156822} using GenerationNode MBM.
[INFO 03-02 10:04:06] ax.api.client: Trial 15 marked COMPLETED.
[INFO 03-02 10:04:08] ax.api.client: Generated new trial 16 with parameters {'w_0': 0.842111, 'w_1': 0.157889} using GenerationNode MBM.
[INFO 03-02 10:04:08] ax.api.client: Trial 16 marked COMPLETED.
[INFO 03-02 10:04:11] ax.api.client: Generated new trial 17 with parameters {'w_0': 0.814581, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 10:04:11] ax.api.client: Trial 17 marked COMPLETED.
[INFO 03-02 10:04:13] ax.api.client: Generated new trial 18 with parameters {'w_0': 0.869114, 'w_1': 0.124058} using GenerationNode MBM.
[INFO 03-02 10:04:13] ax.api.client: Trial 18 marked COMPLETED.
[INFO 03-02 10:04:16] ax.api.client: Generated new trial 19 with parameters {'w_0': 0.861527, 'w_1': 0.0} using GenerationNode MBM.
[INFO 03-02 10:04:16] ax.api.client: Trial 19 marked COMPLETED.
[INFO 03-02 10:04:19] ax.api.client: Generated new trial 20 with parameters {'w_0': 0.422149, 'w_1': 0.577851} using GenerationNode MBM.

GP Bandit - Reward: 831, Trials: 20000

3.5.2.1 Compare Results¶

We’ll plot the results of the experiments.

# Plot accumulated clicks and regret side by side
fig, axes = plt.subplots(1, 2, figsize=(16, 5))

# Accumulated clicks
axes[0].plot(ab_testing_accum_rewards_per_trial, label="AB Testing")
axes[0].plot(ucb1_accum_rewards_per_trial, label="UCB1")
axes[0].plot(
    thompson_sampling_accum_rewards_per_trial, label="Thompson Sampling"
)
axes[0].plot(gp_bandit_accum_rewards_per_trial, label="GP Bandit")
axes[0].set_title("Accumulated Clicks")
axes[0].set_xlabel("Trial")
axes[0].set_ylabel("Accumulated clicks")
axes[0].legend()

# Accumulated regret
axes[1].plot(ab_testing_regret_per_trial, label="AB Testing")
axes[1].plot(ucb1_regret_per_trial, label="UCB1")
axes[1].plot(thompson_sampling_regret_per_trial, label="Thompson Sampling")
axes[1].plot(gp_bandit_regret_per_trial, label="GP Bandit")
axes[1].set_title("Accumulated Regret")
axes[1].set_xlabel("Trial")
axes[1].set_ylabel("Regret")
axes[1].legend()

plt.tight_layout()

Now we’ll plot how each arm was pulled for each algorithm.

# Plot ab_testing_pulls_per_arm_per_trial
fig, axes = plt.subplots(2, 2, figsize=(16, 5))

# AB Testing
for i in range(3):
    axes[0][0].plot(
        ab_testing_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}"
    )
axes[0][0].set_title("AB Testing - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

# UCB1
for i in range(3):
    axes[0][1].plot(ucb1_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}")
axes[0][1].set_title("UCB1 - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

# Thompson Sampling
for i in range(3):
    axes[1][0].plot(
        thompson_sampling_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}"
    )
axes[1][0].set_title("Thompson Sampling - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

# GP Bandit
for i in range(3):
    axes[1][1].plot(
        gp_bandit_pulls_per_arm_per_trial[:, i], label=f"Arm {i + 1}"
    )
axes[1][1].set_title("GP Bandit - Arm pulls per trial")
axes[0][0].set_xlabel("Trial")
axes[0][0].set_ylabel("Accumulated Pulls")
axes[0][0].legend()

plt.tight_layout()

3.5.2.2 Analysis of the results¶

I ran these tests several times, and the results are very similar.

UCB1 doesn’t adapt well in this case where the performance difference between the arms is not very large, and there isn’t one precise “expected CTR” in reality. On top of that, once it has decided what’s the best arm, it will keep exploring the other options without a probability consideration.

Thompson Sampling as expected, gave us the best results. It’s specifically designed for this scenario as it assumes that the CTR is a Beta distribution that will be similar to a Normal distribution after learning, and the CTR is in fact a normal distribution in the simulation.

GP Bandit (Ax) performs pretty well, but it’s not as good as Thompson Sampling. It’s more flexible and as such, it keeps exploring the other options.

Something very important to notice is that looking at the accumulated regret we can see how both Thompson Sampling and GP Bandit converge. They could keep running for a longer time and the regret would keep decreasing.

This is very important, one of the challenges in A/B testing is to know when to stop the experiment, to understand if we’re wasting clicks, or we need to keep trying to be sure we’ve already discovered the best option.