DSP Simulator - pCTR prediction model
This simulation is based on the iPinYou dataset, described in these publications:
- iPinYou Global RTB Bidding Algorithm Competition Dataset (Hairen Liao, Lingxiao Peng, Zhenchuan Liu, Xuehua Shen)
- Real-Time Bidding Benchmarking with iPinYou Dataset (Weinan Zhang, Shuai Yuan, Jun Wang)
For more information:
- iPinYou Global RTB Bidding Algorithm Competition
- Github - wnzhang/make-ipinyou-data
- Github - wnzhang/optimal-rtb
iPinYou is a DSP (Demand Side Platform) for RTB (Real-Time Bidding) in China that published in 2014 a complete dataset of individual bids, impressions and clicks for a certain group of advertisers during a week.
iPinYou opened this dataset for a competition to create the best bidding algorithm. A bidding algorithm is a function that, based on some information given by the Ad Exchange, decides whether to bid for an impression and the bid amount.
There are different algorithms that can be used to decide the bid amount, but all of them require first to estimate the probability of click (pCTR) of an impression.
For this reason, in order to create a simulation of the DSP, which is required to use the Bayesian Optimization technique, first we need to create a pCTR prediction model.
The literature reference above discuss different models to predict the pCTR, but this prediction is done with all the information available. In this case we’re trying to simulate an Adaptive Experiment where the pCTR is predicted based on the available information at the moment of the bid. This means, on June 7th, we only have information from June 6th to predict the pCTR of the bids of June 7th.
At the end of this process we’ll obtain a model file that can be used to obtain the pCTR that a DSP would use to decide whether to bid for an impression and the bid amount.
The Jupyter Notebook requires the “ipinyou.contest.dataset” to create the models, which weighs several GBs. For this reason the dataset is not included in the repository.
This Notebook also generates a cleaner dataset with feature engineering done that can be used to simulate the DSP.
Prepare pCTR prediction model and dataset¶
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score
import xgboost as xgb
import joblib
import globExtract dataset for advertiser 1458¶
Decompress the training2 dataset and filter the rows containing the advertiser id 1458. This reduces the dataset from 21 to 5.4 GB.
Then I use AWK to parse the files and obtain the ones where the Advertiser ID is 1458 (Column 20)
# !bunzip2 ipinyou.contest.dataset/training2nd/*.txt.bz2
# ! for file in ipinyou.contest.dataset/training2nd/*.txt; do grep $'\t1458\t' "$file" > "${file%.txt}_1458.txt"; done
# ! for file in ipinyou.contest.dataset/training2nd/bid.*_1458.txt; do awk -F'\t' '$20 ~ /1458/ {print}' "$file" > "${file/_1458.txt/_1458.tsv}"; done
# ! for file in ipinyou.contest.dataset/training2nd/imp.*_1458.txt; do awk -F'\t' '$23 ~ /1458/ {print}' "$file" > "${file/_1458.txt/_1458.tsv}"; done
# ! for file in ipinyou.contest.dataset/training2nd/clk.*_1458.txt; do awk -F'\t' '$23 ~ /1458/ {print}' "$file" > "${file/_1458.txt/_1458.tsv}"; done# Load the tab separated file in ./ipinyou.contest.txt/training2nd
df_bids = pd.read_csv(
"ipinyou.contest.dataset/training2nd/bid.20130606_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_imps = pd.read_csv(
"ipinyou.contest.dataset/training2nd/imp.20130606_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_clks = pd.read_csv(
"ipinyou.contest.dataset/training2nd/clk.20130606_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)# Rename the columns
df_bids.columns = [
"bid_id",
"timestamp",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"advertiser_id",
"user_profile_ids",
]
# Drop the user_profile_ids column as it's always empty
df_bids = df_bids.drop(columns=["user_profile_ids"])
df_bids.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3250536 entries, 0 to 3250535
Data columns (total 20 columns):
# Column Dtype
--- ------ -----
0 bid_id object
1 timestamp int64
2 ipinyou_id object
3 user_agent object
4 ip_address object
5 region_id int64
6 city_id int64
7 ad_exchange_id int64
8 domain object
9 url object
10 anon_url object
11 slot_id object
12 slot_width int64
13 slot_height int64
14 slot_visibility int64
15 slot_format int64
16 slot_floor_price int64
17 creative_id object
18 bidding_price int64
19 advertiser_id int64
dtypes: int64(11), object(9)
memory usage: 496.0+ MB
df_imps.columns = [
"bid_id",
"timestamp",
"log_type",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"pay_price",
"landing_page_url",
"advertiser_id",
"user_profile_ids",
]
df_imps.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 448164 entries, 0 to 448163
Data columns (total 24 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 bid_id 448164 non-null object
1 timestamp 448164 non-null int64
2 log_type 448164 non-null int64
3 ipinyou_id 448164 non-null object
4 user_agent 448101 non-null object
5 ip_address 448164 non-null object
6 region_id 448164 non-null int64
7 city_id 448164 non-null int64
8 ad_exchange_id 448164 non-null int64
9 domain 422815 non-null object
10 url 438769 non-null object
11 anon_url 0 non-null float64
12 slot_id 448164 non-null object
13 slot_width 448164 non-null int64
14 slot_height 448164 non-null int64
15 slot_visibility 448164 non-null int64
16 slot_format 448164 non-null int64
17 slot_floor_price 448164 non-null int64
18 creative_id 448164 non-null object
19 bidding_price 448164 non-null int64
20 pay_price 448164 non-null int64
21 landing_page_url 448164 non-null object
22 advertiser_id 448164 non-null int64
23 user_profile_ids 393497 non-null object
dtypes: float64(1), int64(13), object(10)
memory usage: 82.1+ MB
df_clks.columns = [
"bid_id",
"timestamp",
"log_type",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"pay_price",
"landing_page_url",
"advertiser_id",
"user_profile_ids",
]
df_clks.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 347 entries, 0 to 346
Data columns (total 24 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 bid_id 347 non-null object
1 timestamp 347 non-null int64
2 log_type 347 non-null int64
3 ipinyou_id 347 non-null object
4 user_agent 347 non-null object
5 ip_address 347 non-null object
6 region_id 347 non-null int64
7 city_id 347 non-null int64
8 ad_exchange_id 347 non-null int64
9 domain 322 non-null object
10 url 328 non-null object
11 anon_url 0 non-null float64
12 slot_id 347 non-null object
13 slot_width 347 non-null int64
14 slot_height 347 non-null int64
15 slot_visibility 347 non-null int64
16 slot_format 347 non-null int64
17 slot_floor_price 347 non-null int64
18 creative_id 347 non-null object
19 bidding_price 347 non-null int64
20 pay_price 347 non-null int64
21 landing_page_url 347 non-null object
22 advertiser_id 347 non-null int64
23 user_profile_ids 278 non-null object
dtypes: float64(1), int64(13), object(10)
memory usage: 65.2+ KB
# From df_imps keep only "bid_id", "pay_price", "user_profile_ids"
df_imps = df_imps[["bid_id", "pay_price", "user_profile_ids"]]
# From df_clks keep only "bid_id". I'm only interested in the clicks
df_clks = df_clks[["bid_id"]]
# Merge the impressions and clicks, add "clicked" column as 1 if matched, 0 otherwise
df_imp_with_clk = pd.merge(
df_imps,
df_clks,
on="bid_id",
how="left",
indicator=True,
suffixes=("_imp", "_clk"),
)
df_imp_with_clk["clicked"] = (df_imp_with_clk["_merge"] == "both").astype(int)
df_imp_with_clk = df_imp_with_clk.drop(columns=["_merge"])
# Free the memory
del df_clks, df_imps
df_imp_with_clk.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 448184 entries, 0 to 448183
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 bid_id 448184 non-null object
1 pay_price 448184 non-null int64
2 user_profile_ids 393510 non-null object
3 clicked 448184 non-null int64
dtypes: int64(2), object(2)
memory usage: 13.7+ MB
df_imp_with_clk["clicked"].value_counts()clicked
0 447836
1 348
Name: count, dtype: int64# Now merge the bids with the impressions
# We only get those who are in both, we're not interested on getting the bids without impressions for the CTR prediction
df_bids_imp_with_clk = pd.merge(
df_bids, df_imp_with_clk, on="bid_id", how="inner", suffixes=("_bid", "_imp")
)
df_bids_imp_with_clk.info()
# Free the memory
del df_bids, df_imp_with_clk<class 'pandas.core.frame.DataFrame'>
RangeIndex: 448184 entries, 0 to 448183
Data columns (total 23 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 bid_id 448184 non-null object
1 timestamp 448184 non-null int64
2 ipinyou_id 298044 non-null object
3 user_agent 448122 non-null object
4 ip_address 448162 non-null object
5 region_id 448184 non-null int64
6 city_id 448184 non-null int64
7 ad_exchange_id 448184 non-null int64
8 domain 422833 non-null object
9 url 438787 non-null object
10 anon_url 15937 non-null object
11 slot_id 448184 non-null object
12 slot_width 448184 non-null int64
13 slot_height 448184 non-null int64
14 slot_visibility 448184 non-null int64
15 slot_format 448184 non-null int64
16 slot_floor_price 448184 non-null int64
17 creative_id 448184 non-null object
18 bidding_price 448184 non-null int64
19 advertiser_id 448184 non-null int64
20 pay_price 448184 non-null int64
21 user_profile_ids 393510 non-null object
22 clicked 448184 non-null int64
dtypes: int64(13), object(10)
memory usage: 78.6+ MB
print("Region IDs:")
print(df_bids_imp_with_clk["region_id"].unique())
print("City IDs:")
print(df_bids_imp_with_clk["city_id"].unique())
print("Ad Exchange IDs:")
print(df_bids_imp_with_clk["ad_exchange_id"].unique())
print("Slot IDs:")
print(df_bids_imp_with_clk["slot_id"].unique())
print("Slot Formats:")
print(df_bids_imp_with_clk["slot_format"].unique())
print("Slot Visibilities:")
print(df_bids_imp_with_clk["slot_visibility"].unique())
print("Slot Widths:")
print(df_bids_imp_with_clk["slot_width"].unique())
print("Slot Heights:")
print(df_bids_imp_with_clk["slot_height"].unique())
print("Creative IDs:")
print(df_bids_imp_with_clk["creative_id"].unique())
print("User Profile IDs:")
print(df_bids_imp_with_clk["user_profile_ids"].unique())Region IDs:
[216 134 368 2 1 124 94 15 164 308 276 65 80 238 333 393 79 201
40 55 394 374 3 0 253 359 106 183 27 146 344 275 298 395 325 -1]
City IDs:
[219 137 369 2 1 125 98 26 171 321 95 224 16 225 290 217 70 83
239 334 233 101 393 227 23 79 202 43 84 57 222 333 81 394 41 378
100 4 0 277 46 177 254 361 80 206 50 338 82 110 380 184 32 97
135 106 186 160 350 375 223 281 149 241 355 174 133 309 91 147 138 113
96 286 129 247 85 314 165 399 275 19 337 231 214 118 132 145 360 250
303 67 294 131 56 37 9 307 299 221 364 395 386 228 146 173 7 59
126 148 152 384 170 42 320 104 151 220 108 203 22 28 178 236 115 94
169 51 24 205 120 172 162 285 190 218 8 237 313 345 139 226 73 15
103 93 211 234 66 144 150 397 300 342 86 212 40 92 213 288 17 196
181 175 31 332 207 245 193 376 167 388 62 305 216 25 127 12 201 232
123 71 114 367 306 153 283 248 77 88 141 235 230 29 240 130 155 243
359 210 105 124 287 188 387 35 302 107 242 189 353 75 13 161 301 27
195 246 112 128 68 176 47 116 187 48 90 158 192 156 5 244 136 180
336 122 372 154 117 53 60 340 278 89 163 39 87 64 109 381 276 315
291 385 52 368 179 49 310 316 215 168 121 34 282 209 204 352 143 159
183 382 111 249 194 208 351 185 311 191 324 6 3 99 65 317 297 45
166 319 54 18 341 36 119 102 293 238 14 20 323 373 344 398 252 280
279 142 379 318 348 346 58 44 370 339 284 343 371 377 140 349 304 331
363 289 326 164 33 229 296 69 354 134 329 21 308 38 76 295 30 72
312 63 255 78 365 383 292 -1 11 251 356 10 157 253 322 347 61 362
335 358 74 325 357 374 327 366 328 330 396]
Ad Exchange IDs:
[1 3 2]
Slot IDs:
['mm_34022157_3445226_11175096' 'mm_10029307_121417_10790029'
'mm_12987374_1803472_13162557' ... '2253353868' '2376324903' '766600122']
Slot Formats:
[1 0]
Slot Visibilities:
[ 2 0 1 255]
Slot Widths:
[ 336 728 160 960 1000 300 250 468]
Slot Heights:
[280 90 600 250 60]
Creative IDs:
['77819d3e0b3467fe5c7b16d68ad923a1' '48f2e9ba15708c0146bda5e1dd653caa'
'cb7c76e7784031272e37af8e7e9b062c' 'fb5afa9dba1274beaf3dad86baf97e89'
'832b91d59d0cb5731431653204a76c0e' 'a499988a822facd86dd0e8e4ffef8532'
'2abc9eaf57d17a96195af3f63c45dc72' '612599432d200b093719dd1f372f7a30']
User Profile IDs:
['10006,10110' '13042,10006,13866,10031,10110'
'10074,10083,13800,10077,10006,10063,10075,10048,10031,10059,10076,10024,10079,10057,10111'
...
'10024,10057,10006,10075,14273,10063,10079,10052,13800,10083,13776,10093,10110'
'13042,10031,14273,10063,10006,10057,10110'
'10006,10057,10063,13042,10031,13866,10110']
Feature Engineering¶
# Transform timestamp to datetime
df_bids_imp_with_clk["timestamp"] = pd.to_datetime(
df_bids_imp_with_clk["timestamp"], errors="coerce"
)
# Create a new column with the hour of the day
df_bids_imp_with_clk["hour"] = df_bids_imp_with_clk["timestamp"].dt.hour
# Create a new column with the day of the week
df_bids_imp_with_clk["day_of_week"] = df_bids_imp_with_clk[
"timestamp"
].dt.dayofweek
def get_browser(user_agent):
if pd.isna(user_agent):
return "Unknown"
ua = user_agent.lower()
if "chrome" in ua:
return "Chrome"
elif "firefox" in ua:
return "Firefox"
elif "safari" in ua and "chrome" not in ua:
return "Safari"
elif "msie" in ua or "trident" in ua:
return "Explorer"
elif "edge" in ua:
return "Edge"
elif "opera" in ua:
return "Opera"
else:
return "Other"
def get_os(user_agent):
if pd.isna(user_agent):
return "Unknown"
ua = user_agent.lower()
if "windows" in ua:
return "Windows"
elif "mac os x" in ua:
return "MacOS"
elif "linux" in ua:
return "Linux"
elif "android" in ua:
return "Android"
elif "iphone" in ua or "ios" in ua:
return "iOS"
else:
return "Other"
# Create new columns for browser and operating system
df_bids_imp_with_clk["browser"] = df_bids_imp_with_clk["user_agent"].apply(
get_browser
)
df_bids_imp_with_clk["os"] = df_bids_imp_with_clk["user_agent"].apply(get_os)
df_bids_imp_with_clk.head()Loading...
# Create one-hot encoding
one_hot_cols = [
"browser",
"os",
"region_id",
"ad_exchange_id",
"slot_visibility",
"slot_format",
"slot_width",
"slot_height",
"creative_id",
]
df_bids_imp_with_clk = pd.get_dummies(
df_bids_imp_with_clk, columns=one_hot_cols, prefix=one_hot_cols
)
# Convert boolean hot-encoded columns to int
for col in df_bids_imp_with_clk.columns:
if df_bids_imp_with_clk[col].dtype == "bool":
df_bids_imp_with_clk[col] = df_bids_imp_with_clk[col].astype(int)
df_bids_imp_with_clk.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 448184 entries, 0 to 448183
Data columns (total 98 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 bid_id 448184 non-null object
1 timestamp 448184 non-null datetime64[ns]
2 ipinyou_id 298044 non-null object
3 user_agent 448122 non-null object
4 ip_address 448162 non-null object
5 city_id 448184 non-null int64
6 domain 422833 non-null object
7 url 438787 non-null object
8 anon_url 15937 non-null object
9 slot_id 448184 non-null object
10 slot_floor_price 448184 non-null int64
11 bidding_price 448184 non-null int64
12 advertiser_id 448184 non-null int64
13 pay_price 448184 non-null int64
14 user_profile_ids 393510 non-null object
15 clicked 448184 non-null int64
16 hour 448184 non-null int32
17 day_of_week 448184 non-null int32
18 browser_Chrome 448184 non-null int64
19 browser_Explorer 448184 non-null int64
20 browser_Firefox 448184 non-null int64
21 browser_Opera 448184 non-null int64
22 browser_Other 448184 non-null int64
23 browser_Safari 448184 non-null int64
24 browser_Unknown 448184 non-null int64
25 os_Android 448184 non-null int64
26 os_Linux 448184 non-null int64
27 os_MacOS 448184 non-null int64
28 os_Other 448184 non-null int64
29 os_Unknown 448184 non-null int64
30 os_Windows 448184 non-null int64
31 os_iOS 448184 non-null int64
32 region_id_-1 448184 non-null int64
33 region_id_0 448184 non-null int64
34 region_id_1 448184 non-null int64
35 region_id_2 448184 non-null int64
36 region_id_3 448184 non-null int64
37 region_id_15 448184 non-null int64
38 region_id_27 448184 non-null int64
39 region_id_40 448184 non-null int64
40 region_id_55 448184 non-null int64
41 region_id_65 448184 non-null int64
42 region_id_79 448184 non-null int64
43 region_id_80 448184 non-null int64
44 region_id_94 448184 non-null int64
45 region_id_106 448184 non-null int64
46 region_id_124 448184 non-null int64
47 region_id_134 448184 non-null int64
48 region_id_146 448184 non-null int64
49 region_id_164 448184 non-null int64
50 region_id_183 448184 non-null int64
51 region_id_201 448184 non-null int64
52 region_id_216 448184 non-null int64
53 region_id_238 448184 non-null int64
54 region_id_253 448184 non-null int64
55 region_id_275 448184 non-null int64
56 region_id_276 448184 non-null int64
57 region_id_298 448184 non-null int64
58 region_id_308 448184 non-null int64
59 region_id_325 448184 non-null int64
60 region_id_333 448184 non-null int64
61 region_id_344 448184 non-null int64
62 region_id_359 448184 non-null int64
63 region_id_368 448184 non-null int64
64 region_id_374 448184 non-null int64
65 region_id_393 448184 non-null int64
66 region_id_394 448184 non-null int64
67 region_id_395 448184 non-null int64
68 ad_exchange_id_1 448184 non-null int64
69 ad_exchange_id_2 448184 non-null int64
70 ad_exchange_id_3 448184 non-null int64
71 slot_visibility_0 448184 non-null int64
72 slot_visibility_1 448184 non-null int64
73 slot_visibility_2 448184 non-null int64
74 slot_visibility_255 448184 non-null int64
75 slot_format_0 448184 non-null int64
76 slot_format_1 448184 non-null int64
77 slot_width_160 448184 non-null int64
78 slot_width_250 448184 non-null int64
79 slot_width_300 448184 non-null int64
80 slot_width_336 448184 non-null int64
81 slot_width_468 448184 non-null int64
82 slot_width_728 448184 non-null int64
83 slot_width_960 448184 non-null int64
84 slot_width_1000 448184 non-null int64
85 slot_height_60 448184 non-null int64
86 slot_height_90 448184 non-null int64
87 slot_height_250 448184 non-null int64
88 slot_height_280 448184 non-null int64
89 slot_height_600 448184 non-null int64
90 creative_id_2abc9eaf57d17a96195af3f63c45dc72 448184 non-null int64
91 creative_id_48f2e9ba15708c0146bda5e1dd653caa 448184 non-null int64
92 creative_id_612599432d200b093719dd1f372f7a30 448184 non-null int64
93 creative_id_77819d3e0b3467fe5c7b16d68ad923a1 448184 non-null int64
94 creative_id_832b91d59d0cb5731431653204a76c0e 448184 non-null int64
95 creative_id_a499988a822facd86dd0e8e4ffef8532 448184 non-null int64
96 creative_id_cb7c76e7784031272e37af8e7e9b062c 448184 non-null int64
97 creative_id_fb5afa9dba1274beaf3dad86baf97e89 448184 non-null int64
dtypes: datetime64[ns](1), int32(2), int64(86), object(9)
memory usage: 331.7+ MB
# List of features as specified
features = ["hour", "day_of_week"]
# Dynamically add all columns that start with the specified prefixes
prefixes = [
"browser_",
"os_",
"ad_exchange_id_",
"slot_visibility_",
"slot_format_",
"slot_width_",
"slot_height_",
"creative_id_",
]
dynamic_features = []
for prefix in prefixes:
dynamic_features.extend(
[col for col in df_bids_imp_with_clk.columns if col.startswith(prefix)]
)
features.extend(dynamic_features)
# Drop rows with missing required feature columns, if any
initial_row_count = len(df_bids_imp_with_clk)
df = df_bids_imp_with_clk.dropna(subset=features + ["clicked"])
dropped_rows = initial_row_count - len(df)
print(f"Rows dropped due to missing features or clicked label: {dropped_rows}")
# Prepare X and y
X = df[features]
y = df["clicked"]
# Split data into train and test
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Train an XGBoost model for binary classification
xgb_model = xgb.XGBClassifier(
n_estimators=100,
max_depth=5,
learning_rate=0.1,
subsample=0.8,
colsample_bytree=0.8,
random_state=42,
eval_metric="logloss",
)
xgb_model.fit(X_train, y_train)
# Predict on test set
y_pred = xgb_model.predict(X_test)
y_pred_proba = xgb_model.predict_proba(X_test)[:, 1]
# Display results
print(classification_report(y_test, y_pred, zero_division=0))
print("ROC AUC:", roc_auc_score(y_test, y_pred_proba))Rows dropped due to missing features or clicked label: 0
precision recall f1-score support
0 1.00 1.00 1.00 89567
1 0.00 0.00 0.00 70
accuracy 1.00 89637
macro avg 0.50 0.50 0.50 89637
weighted avg 1.00 1.00 1.00 89637
ROC AUC: 0.6486301555579304
Create CTR Prediction Model with XGBoost¶
y_pred_proba.mean()np.float32(0.00078361767)# Store the trained XGBoost model in a file
joblib.dump(xgb_model, "xgb_model.joblib")['xgb_model.joblib']# Load the trained model
loaded_model = joblib.load("xgb_model.joblib")
# Select a few records from the original dataset for inference
X_few = X_test.head(100)
# Predict probabilities and classes for these records
few_pred_proba = loaded_model.predict_proba(X_few)[:, 1]
few_pred = loaded_model.predict(X_few)
# Display the results
inference_df = X_few.copy()
inference_df["predicted_clicked"] = few_pred
inference_df["predicted_proba"] = few_pred_proba
print(inference_df[["predicted_clicked", "predicted_proba"]])
print(y_pred_proba.mean())
print("ROC AUC:", roc_auc_score(y_test, y_pred_proba)) predicted_clicked predicted_proba
6289 0 0.000214
430694 0 0.000389
290441 0 0.001132
36148 0 0.000326
173864 0 0.000891
... ... ...
218279 0 0.001027
92209 0 0.000951
354769 0 0.000738
345183 0 0.000216
109306 0 0.000951
[100 rows x 2 columns]
0.00078361767
ROC AUC: 0.6486301555579304
The ROC AUC is better than random. The mean predicted probability of click is 0.0007, the same than the average click rate of the advertiser. This is a good sign.
Train models for each day¶
In the experiment we’re going to simulate, we need to estimate the CTR given a Bid. The dataset contains bids, impressions and clicks data from June 6th to June 12th.
The simulation will start on day 2 (June 7th) and run the bids registered on that day. At that point we should only have access to the previous day’s data to predict the CTR.
I’m going to create 6 different models:
- June 6th -> To use on June 7th
- June 6th and 7th -> To use on June 8th
- June 6th to 8th -> To use on June 9th
- June 6th to 9th -> To use on June 10th
- June 6th to 10th -> To use on June 11th
- June 6th to 11th -> To use on June 12th
The code to create the models will be packaged as a function to be called multiple times.
TRAINING_DATA_DIR = "ipinyou.contest.dataset/training2nd"
def get_browser(user_agent):
if pd.isna(user_agent):
return "Unknown"
ua = user_agent.lower()
if "chrome" in ua:
return "Chrome"
elif "firefox" in ua:
return "Firefox"
elif "safari" in ua and "chrome" not in ua:
return "Safari"
elif "msie" in ua or "trident" in ua:
return "Explorer"
elif "edge" in ua:
return "Edge"
elif "opera" in ua:
return "Opera"
else:
return "Other"
def get_os(user_agent):
if pd.isna(user_agent):
return "Unknown"
ua = user_agent.lower()
if "windows" in ua:
return "Windows"
elif "mac os x" in ua:
return "MacOS"
elif "linux" in ua:
return "Linux"
elif "android" in ua:
return "Android"
elif "iphone" in ua or "ios" in ua:
return "iOS"
else:
return "Other"
def create_model(dates: list[str]):
print("====================================================")
print(f"Create prediction model from {dates[0]} to {dates[-1]}")
print("====================================================")
all_bids_imp_with_clk = []
for date in dates:
df_bids = pd.read_csv(
f"{TRAINING_DATA_DIR}/bid.{date}_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_imps = pd.read_csv(
f"{TRAINING_DATA_DIR}/imp.{date}_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_clks = pd.read_csv(
f"{TRAINING_DATA_DIR}/clk.{date}_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_bids.columns = [
"bid_id",
"timestamp",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"advertiser_id",
"user_profile_ids",
]
df_bids = df_bids.drop(columns=["user_profile_ids"])
df_imps.columns = [
"bid_id",
"timestamp",
"log_type",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"pay_price",
"landing_page_url",
"advertiser_id",
"user_profile_ids",
]
df_clks.columns = [
"bid_id",
"timestamp",
"log_type",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"pay_price",
"landing_page_url",
"advertiser_id",
"user_profile_ids",
]
df_imps = df_imps[["bid_id", "pay_price", "user_profile_ids"]]
df_clks = df_clks[["bid_id"]]
# Merge the impressions and clicks, add "clicked" column as 1 if matched, 0 otherwise
df_imp_with_clk = pd.merge(
df_imps,
df_clks,
on="bid_id",
how="left",
indicator=True,
suffixes=("_imp", "_clk"),
)
df_imp_with_clk["clicked"] = (
df_imp_with_clk["_merge"] == "both"
).astype(int)
df_imp_with_clk = df_imp_with_clk.drop(columns=["_merge"])
del df_clks, df_imps
# Now merge the bids with the impressions
# We only get those who are in both, we're not interested on getting the bids without impressions for the CTR prediction
df_bids_imp_with_clk = pd.merge(
df_bids,
df_imp_with_clk,
on="bid_id",
how="inner",
suffixes=("_bid", "_imp"),
)
del df_bids, df_imp_with_clk
# Feature Engineering
df_bids_imp_with_clk["timestamp"] = pd.to_datetime(
df_bids_imp_with_clk["timestamp"], errors="coerce"
)
df_bids_imp_with_clk["hour"] = df_bids_imp_with_clk["timestamp"].dt.hour
df_bids_imp_with_clk["day_of_week"] = df_bids_imp_with_clk[
"timestamp"
].dt.dayofweek
# Create new columns for browser and operating system
df_bids_imp_with_clk["browser"] = df_bids_imp_with_clk[
"user_agent"
].apply(get_browser)
df_bids_imp_with_clk["os"] = df_bids_imp_with_clk["user_agent"].apply(
get_os
)
# Create one-hot encoding
one_hot_cols = [
"browser",
"os",
"region_id",
"ad_exchange_id",
"slot_visibility",
"slot_format",
"slot_width",
"slot_height",
"creative_id",
]
df_bids_imp_with_clk = pd.get_dummies(
df_bids_imp_with_clk, columns=one_hot_cols, prefix=one_hot_cols
)
# Convert boolean hot-encoded columns to int
for col in df_bids_imp_with_clk.columns:
if df_bids_imp_with_clk[col].dtype == "bool":
df_bids_imp_with_clk[col] = df_bids_imp_with_clk[col].astype(int)
# Append for later concatenation
all_bids_imp_with_clk.append(df_bids_imp_with_clk)
# Concatenate all DataFrames into a single DataFrame
df_bids_imp_with_clk = pd.concat(all_bids_imp_with_clk, ignore_index=True)
avg_spent_per_day = df_bids_imp_with_clk["pay_price"].sum() / len(dates)
# Start training the XGBoost model
features = ["hour", "day_of_week"]
# Define the columns dynamically based on the one-hot encoding
prefixes = [
"browser_",
"os_",
"ad_exchange_id_",
"slot_visibility_",
"slot_format_",
"slot_width_",
"slot_height_",
"creative_id_",
]
dynamic_features = []
for prefix in prefixes:
dynamic_features.extend(
[
col
for col in df_bids_imp_with_clk.columns
if col.startswith(prefix)
]
)
features.extend(dynamic_features)
print(
f"Training CTR prediction model from {dates[0]} to {dates[-1]} with features: {features}"
)
# Drop rows with missing required feature columns, if any
initial_row_count = len(df_bids_imp_with_clk)
df = df_bids_imp_with_clk.dropna(subset=features + ["clicked"])
dropped_rows = initial_row_count - len(df)
print(
f"Rows dropped due to missing features or clicked label: {dropped_rows}"
)
# Prepare X and y
X = df[features]
y = df["clicked"]
# Split data into train and test
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Train an XGBoost model for binary classification
xgb_model = xgb.XGBClassifier(
n_estimators=100,
max_depth=5,
learning_rate=0.1,
subsample=0.8,
colsample_bytree=0.8,
random_state=42,
eval_metric="logloss",
)
xgb_model.fit(X_train, y_train)
# Predict on test set
y_pred = xgb_model.predict(X_test)
y_pred_proba = xgb_model.predict_proba(X_test)[:, 1]
# Display results
print(classification_report(y_test, y_pred, zero_division=0))
print("ROC AUC:", roc_auc_score(y_test, y_pred_proba))
print(f"Avg ground truth CTR: {y_test.mean()}")
print(f"Avg predicted CTR: {y_pred_proba.mean()}")
# Store the model in a file
joblib.dump(xgb_model, f"xgb_model_{dates[-1]}.joblib")
# Store the features used by this model in a file
with open(f"features_{dates[-1]}.txt", "w") as f:
f.write("\n".join(features))
# Store the average CTR and the bid to pay ratio
average_ctr = df["clicked"].mean()
bid_to_pay_ratio = df["bidding_price"].mean() / df["pay_price"].mean()
# Store average_ctr and bid_to_pay_ratio in a file
import json
with open(f"average_ctr_and_bid_to_pay_ratio_{dates[-1]}.json", "w") as f:
json.dump(
{
"average_ctr": average_ctr,
"bid_to_pay_ratio": bid_to_pay_ratio,
"avg_spent_per_day": avg_spent_per_day,
},
f,
indent=4,
)
return xgb_model, average_ctr, bid_to_pay_ratiocreate_model(["20130606"])
create_model(["20130606", "20130607"])
create_model(["20130606", "20130607", "20130608"])
create_model(["20130606", "20130607", "20130608", "20130609"])
create_model(["20130606", "20130607", "20130608", "20130609", "20130610"])
create_model(
["20130606", "20130607", "20130608", "20130609", "20130610", "20130611"]
)====================================================
Create prediction model from 20130606 to 20130606
====================================================
Training CTR prediction model from 20130606 to 20130606 with features: ['hour', 'day_of_week', 'browser_Chrome', 'browser_Explorer', 'browser_Firefox', 'browser_Opera', 'browser_Other', 'browser_Safari', 'browser_Unknown', 'os_Android', 'os_Linux', 'os_MacOS', 'os_Other', 'os_Unknown', 'os_Windows', 'os_iOS', 'ad_exchange_id_1', 'ad_exchange_id_2', 'ad_exchange_id_3', 'slot_visibility_0', 'slot_visibility_1', 'slot_visibility_2', 'slot_visibility_255', 'slot_format_0', 'slot_format_1', 'slot_width_160', 'slot_width_250', 'slot_width_300', 'slot_width_336', 'slot_width_468', 'slot_width_728', 'slot_width_960', 'slot_width_1000', 'slot_height_60', 'slot_height_90', 'slot_height_250', 'slot_height_280', 'slot_height_600', 'creative_id_2abc9eaf57d17a96195af3f63c45dc72', 'creative_id_48f2e9ba15708c0146bda5e1dd653caa', 'creative_id_612599432d200b093719dd1f372f7a30', 'creative_id_77819d3e0b3467fe5c7b16d68ad923a1', 'creative_id_832b91d59d0cb5731431653204a76c0e', 'creative_id_a499988a822facd86dd0e8e4ffef8532', 'creative_id_cb7c76e7784031272e37af8e7e9b062c', 'creative_id_fb5afa9dba1274beaf3dad86baf97e89']
Rows dropped due to missing features or clicked label: 0
precision recall f1-score support
0 1.00 1.00 1.00 89567
1 0.00 0.00 0.00 70
accuracy 1.00 89637
macro avg 0.50 0.50 0.50 89637
weighted avg 1.00 1.00 1.00 89637
ROC AUC: 0.6486301555579304
Avg ground truth CTR: 0.0007809275187701507
Avg predicted CTR: 0.0007836176664568484
====================================================
Create prediction model from 20130606 to 20130607
====================================================
Training CTR prediction model from 20130606 to 20130607 with features: ['hour', 'day_of_week', 'browser_Chrome', 'browser_Explorer', 'browser_Firefox', 'browser_Opera', 'browser_Other', 'browser_Safari', 'browser_Unknown', 'os_Android', 'os_Linux', 'os_MacOS', 'os_Other', 'os_Unknown', 'os_Windows', 'os_iOS', 'ad_exchange_id_1', 'ad_exchange_id_2', 'ad_exchange_id_3', 'slot_visibility_0', 'slot_visibility_1', 'slot_visibility_2', 'slot_visibility_255', 'slot_format_0', 'slot_format_1', 'slot_width_160', 'slot_width_250', 'slot_width_300', 'slot_width_336', 'slot_width_468', 'slot_width_728', 'slot_width_960', 'slot_width_1000', 'slot_height_60', 'slot_height_90', 'slot_height_250', 'slot_height_280', 'slot_height_600', 'creative_id_2abc9eaf57d17a96195af3f63c45dc72', 'creative_id_48f2e9ba15708c0146bda5e1dd653caa', 'creative_id_612599432d200b093719dd1f372f7a30', 'creative_id_77819d3e0b3467fe5c7b16d68ad923a1', 'creative_id_832b91d59d0cb5731431653204a76c0e', 'creative_id_a499988a822facd86dd0e8e4ffef8532', 'creative_id_cb7c76e7784031272e37af8e7e9b062c', 'creative_id_fb5afa9dba1274beaf3dad86baf97e89']
Rows dropped due to missing features or clicked label: 0
precision recall f1-score support
0 1.00 1.00 1.00 185128
1 0.00 0.00 0.00 135
accuracy 1.00 185263
macro avg 0.50 0.50 0.50 185263
weighted avg 1.00 1.00 1.00 185263
ROC AUC: 0.6413705152150985
Avg ground truth CTR: 0.0007286938028640365
Avg predicted CTR: 0.0007366804638877511
====================================================
Create prediction model from 20130606 to 20130608
====================================================
Training CTR prediction model from 20130606 to 20130608 with features: ['hour', 'day_of_week', 'browser_Chrome', 'browser_Explorer', 'browser_Firefox', 'browser_Opera', 'browser_Other', 'browser_Safari', 'browser_Unknown', 'os_Android', 'os_Linux', 'os_MacOS', 'os_Other', 'os_Unknown', 'os_Windows', 'os_iOS', 'ad_exchange_id_1', 'ad_exchange_id_2', 'ad_exchange_id_3', 'slot_visibility_0', 'slot_visibility_1', 'slot_visibility_2', 'slot_visibility_255', 'slot_format_0', 'slot_format_1', 'slot_width_160', 'slot_width_250', 'slot_width_300', 'slot_width_336', 'slot_width_468', 'slot_width_728', 'slot_width_960', 'slot_width_1000', 'slot_height_60', 'slot_height_90', 'slot_height_250', 'slot_height_280', 'slot_height_600', 'creative_id_2abc9eaf57d17a96195af3f63c45dc72', 'creative_id_48f2e9ba15708c0146bda5e1dd653caa', 'creative_id_612599432d200b093719dd1f372f7a30', 'creative_id_77819d3e0b3467fe5c7b16d68ad923a1', 'creative_id_832b91d59d0cb5731431653204a76c0e', 'creative_id_a499988a822facd86dd0e8e4ffef8532', 'creative_id_cb7c76e7784031272e37af8e7e9b062c', 'creative_id_fb5afa9dba1274beaf3dad86baf97e89']
Rows dropped due to missing features or clicked label: 0
precision recall f1-score support
0 1.00 1.00 1.00 267820
1 0.00 0.00 0.00 209
accuracy 1.00 268029
macro avg 0.50 0.50 0.50 268029
weighted avg 1.00 1.00 1.00 268029
ROC AUC: 0.6745654708457691
Avg ground truth CTR: 0.0007797663685645956
Avg predicted CTR: 0.0007827499648556113
====================================================
Create prediction model from 20130606 to 20130609
====================================================
Training CTR prediction model from 20130606 to 20130609 with features: ['hour', 'day_of_week', 'browser_Chrome', 'browser_Explorer', 'browser_Firefox', 'browser_Opera', 'browser_Other', 'browser_Safari', 'browser_Unknown', 'os_Android', 'os_Linux', 'os_MacOS', 'os_Other', 'os_Unknown', 'os_Windows', 'os_iOS', 'ad_exchange_id_1', 'ad_exchange_id_2', 'ad_exchange_id_3', 'slot_visibility_0', 'slot_visibility_1', 'slot_visibility_2', 'slot_visibility_255', 'slot_format_0', 'slot_format_1', 'slot_width_160', 'slot_width_250', 'slot_width_300', 'slot_width_336', 'slot_width_468', 'slot_width_728', 'slot_width_960', 'slot_width_1000', 'slot_height_60', 'slot_height_90', 'slot_height_250', 'slot_height_280', 'slot_height_600', 'creative_id_2abc9eaf57d17a96195af3f63c45dc72', 'creative_id_48f2e9ba15708c0146bda5e1dd653caa', 'creative_id_612599432d200b093719dd1f372f7a30', 'creative_id_77819d3e0b3467fe5c7b16d68ad923a1', 'creative_id_832b91d59d0cb5731431653204a76c0e', 'creative_id_a499988a822facd86dd0e8e4ffef8532', 'creative_id_cb7c76e7784031272e37af8e7e9b062c', 'creative_id_fb5afa9dba1274beaf3dad86baf97e89']
Rows dropped due to missing features or clicked label: 0
precision recall f1-score support
0 1.00 1.00 1.00 352494
1 0.00 0.00 0.00 283
accuracy 1.00 352777
macro avg 0.50 0.50 0.50 352777
weighted avg 1.00 1.00 1.00 352777
ROC AUC: 0.656206723695129
Avg ground truth CTR: 0.0008022064930536855
Avg predicted CTR: 0.0008045399445109069
====================================================
Create prediction model from 20130606 to 20130610
====================================================
Training CTR prediction model from 20130606 to 20130610 with features: ['hour', 'day_of_week', 'browser_Chrome', 'browser_Explorer', 'browser_Firefox', 'browser_Opera', 'browser_Other', 'browser_Safari', 'browser_Unknown', 'os_Android', 'os_Linux', 'os_MacOS', 'os_Other', 'os_Unknown', 'os_Windows', 'os_iOS', 'ad_exchange_id_1', 'ad_exchange_id_2', 'ad_exchange_id_3', 'slot_visibility_0', 'slot_visibility_1', 'slot_visibility_2', 'slot_visibility_255', 'slot_format_0', 'slot_format_1', 'slot_width_160', 'slot_width_250', 'slot_width_300', 'slot_width_336', 'slot_width_468', 'slot_width_728', 'slot_width_960', 'slot_width_1000', 'slot_height_60', 'slot_height_90', 'slot_height_250', 'slot_height_280', 'slot_height_600', 'creative_id_2abc9eaf57d17a96195af3f63c45dc72', 'creative_id_48f2e9ba15708c0146bda5e1dd653caa', 'creative_id_612599432d200b093719dd1f372f7a30', 'creative_id_77819d3e0b3467fe5c7b16d68ad923a1', 'creative_id_832b91d59d0cb5731431653204a76c0e', 'creative_id_a499988a822facd86dd0e8e4ffef8532', 'creative_id_cb7c76e7784031272e37af8e7e9b062c', 'creative_id_fb5afa9dba1274beaf3dad86baf97e89']
Rows dropped due to missing features or clicked label: 0
precision recall f1-score support
0 1.00 1.00 1.00 439269
1 0.00 0.00 0.00 360
accuracy 1.00 439629
macro avg 0.50 0.50 0.50 439629
weighted avg 1.00 1.00 1.00 439629
ROC AUC: 0.6429470545889244
Avg ground truth CTR: 0.0008188722763966891
Avg predicted CTR: 0.0008188612409867346
====================================================
Create prediction model from 20130606 to 20130611
====================================================
Training CTR prediction model from 20130606 to 20130611 with features: ['hour', 'day_of_week', 'browser_Chrome', 'browser_Explorer', 'browser_Firefox', 'browser_Opera', 'browser_Other', 'browser_Safari', 'browser_Unknown', 'os_Android', 'os_Linux', 'os_MacOS', 'os_Other', 'os_Unknown', 'os_Windows', 'os_iOS', 'ad_exchange_id_1', 'ad_exchange_id_2', 'ad_exchange_id_3', 'slot_visibility_0', 'slot_visibility_1', 'slot_visibility_2', 'slot_visibility_255', 'slot_format_0', 'slot_format_1', 'slot_width_160', 'slot_width_250', 'slot_width_300', 'slot_width_336', 'slot_width_468', 'slot_width_728', 'slot_width_960', 'slot_width_1000', 'slot_height_60', 'slot_height_90', 'slot_height_250', 'slot_height_280', 'slot_height_600', 'creative_id_2abc9eaf57d17a96195af3f63c45dc72', 'creative_id_48f2e9ba15708c0146bda5e1dd653caa', 'creative_id_612599432d200b093719dd1f372f7a30', 'creative_id_77819d3e0b3467fe5c7b16d68ad923a1', 'creative_id_832b91d59d0cb5731431653204a76c0e', 'creative_id_a499988a822facd86dd0e8e4ffef8532', 'creative_id_cb7c76e7784031272e37af8e7e9b062c', 'creative_id_fb5afa9dba1274beaf3dad86baf97e89']
Rows dropped due to missing features or clicked label: 0
precision recall f1-score support
0 1.00 1.00 1.00 526694
1 0.00 0.00 0.00 443
accuracy 1.00 527137
macro avg 0.50 0.50 0.50 527137
weighted avg 1.00 1.00 1.00 527137
ROC AUC: 0.6842191902930157
Avg ground truth CTR: 0.0008403887414467207
Avg predicted CTR: 0.0008393076132051647
(XGBClassifier(base_score=None, booster=None, callbacks=None,
colsample_bylevel=None, colsample_bynode=None,
colsample_bytree=0.8, device=None, early_stopping_rounds=None,
enable_categorical=False, eval_metric='logloss',
feature_types=None, feature_weights=None, gamma=None,
grow_policy=None, importance_type=None,
interaction_constraints=None, learning_rate=0.1, max_bin=None,
max_cat_threshold=None, max_cat_to_onehot=None,
max_delta_step=None, max_depth=5, max_leaves=None,
min_child_weight=None, missing=nan, monotone_constraints=None,
multi_strategy=None, n_estimators=100, n_jobs=None,
num_parallel_tree=None, ...),
np.float64(0.000840769106440003),
np.float64(4.341801088428895))Evaluate predicted CTR¶
Supposing we are on June 12th, we receive a bid, we want to predict the CTR of the bid.
We need to use the model trained on data from June 6th to June 11th, read a line from the June 12th bid file, and extract the features from the bid
LAST_DATE = "20130611"
SIMULATION_DATE = "20130612"
df_bids = pd.read_csv(
f"ipinyou.contest.dataset/training2nd/bid.{SIMULATION_DATE}_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_bids.columns = [
"bid_id",
"timestamp",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"advertiser_id",
"user_profile_ids",
]
# Load the model
model = joblib.load(f"xgb_model_{LAST_DATE}.joblib")
# Load the features
features = open(f"features_{LAST_DATE}.txt", "r").read().splitlines()first_bid = df_bids.iloc[[0]].copy()
def predict_ctr(
bid: pd.DataFrame, model: xgb.XGBClassifier, features: list[str]
):
# Process the fields appropriately on that DataFrame row
bid["timestamp"] = pd.to_datetime(bid["timestamp"], errors="coerce")
bid["hour"] = bid["timestamp"].dt.hour
bid["day_of_week"] = bid["timestamp"].dt.dayofweek
bid["browser"] = bid["user_agent"].apply(get_browser)
bid["os"] = bid["user_agent"].apply(get_os)
# Create one-hot encoding
one_hot_cols = [
"browser",
"os",
"region_id",
"ad_exchange_id",
"slot_visibility",
"slot_format",
"slot_width",
"slot_height",
"creative_id",
]
bid = pd.get_dummies(bid, columns=one_hot_cols, prefix=one_hot_cols)
# Convert boolean hot-encoded columns to int
for col in bid.columns:
if bid[col].dtype == "bool":
bid[col] = bid[col].astype(int)
missing_features = [f for f in features if f not in bid.columns]
for f in missing_features:
bid[f] = False
# Select features in correct order for the model
X_bid = bid[features]
# Predict CTR using the loaded model
predicted_ctr = model.predict_proba(X_bid)[:, 1][0]
print("Predicted CTR for first bid:", predicted_ctr)predict_ctr(first_bid, model, features)
second_bid = df_bids.iloc[[1]].copy()
predict_ctr(second_bid, model, features)
third_bid = df_bids.iloc[[2]].copy()
predict_ctr(third_bid, model, features)
fourth_bid = df_bids.iloc[[3]].copy()
predict_ctr(fourth_bid, model, features)
fifth_bid = df_bids.iloc[[4]].copy()
predict_ctr(fifth_bid, model, features)Predicted CTR for first bid: 0.00031548683
Predicted CTR for first bid: 0.0012850019
Predicted CTR for first bid: 0.0007039644
Predicted CTR for first bid: 0.00031548683
Predicted CTR for first bid: 0.001144583
Create feature engineered dataset¶
The following code creates a new dataset with the feature engineered columns, that can be used to simulate the DSP.
# Read all the feature files to discover all the possible features generated by the one-hot encoding
feature_files = glob.glob("features_*.txt")
features = []
for file in feature_files:
# I don't use a set because I want to keep the order of the features
with open(file, "r") as f:
for line in f:
a_feature = line.strip()
if a_feature not in features:
features.append(a_feature)
average_ctrs = {}
for date in [
"20130606",
"20130607",
"20130608",
"20130609",
"20130610",
"20130611",
"20130612",
]:
df_bids = pd.read_csv(
f"{TRAINING_DATA_DIR}/bid.{date}_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_imps = pd.read_csv(
f"{TRAINING_DATA_DIR}/imp.{date}_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_clks = pd.read_csv(
f"{TRAINING_DATA_DIR}/clk.{date}_1458.tsv",
sep="\t",
header=None,
low_memory=False,
)
df_bids.columns = [
"bid_id",
"timestamp",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"advertiser_id",
"user_profile_ids",
]
df_bids = df_bids.drop(columns=["user_profile_ids"])
df_imps.columns = [
"bid_id",
"timestamp",
"log_type",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"pay_price",
"landing_page_url",
"advertiser_id",
"user_profile_ids",
]
df_clks.columns = [
"bid_id",
"timestamp",
"log_type",
"ipinyou_id",
"user_agent",
"ip_address",
"region_id",
"city_id",
"ad_exchange_id",
"domain",
"url",
"anon_url",
"slot_id",
"slot_width",
"slot_height",
"slot_visibility",
"slot_format",
"slot_floor_price",
"creative_id",
"bidding_price",
"pay_price",
"landing_page_url",
"advertiser_id",
"user_profile_ids",
]
df_imps = df_imps[["bid_id", "pay_price", "user_profile_ids"]]
df_clks = df_clks[["bid_id"]]
# Merge the impressions and clicks, add "clicked" column as 1 if matched, 0 otherwise
df_imp_with_clk = pd.merge(
df_imps,
df_clks,
on="bid_id",
how="left",
indicator=True,
suffixes=("_imp", "_clk"),
)
df_imp_with_clk["clicked"] = (df_imp_with_clk["_merge"] == "both").astype(
int
)
df_imp_with_clk = df_imp_with_clk.drop(columns=["_merge"])
del df_clks, df_imps
# Now merge the bids with the impressions
# We only get those who are in both, we're not interested on getting the bids without impressions for the CTR prediction
df_bids_imp_with_clk = pd.merge(
df_bids,
df_imp_with_clk,
on="bid_id",
how="inner",
suffixes=("_bid", "_imp"),
)
del df_bids, df_imp_with_clk
# Feature Engineering
df_bids_imp_with_clk["timestamp"] = pd.to_datetime(
df_bids_imp_with_clk["timestamp"], errors="coerce"
)
df_bids_imp_with_clk["hour"] = df_bids_imp_with_clk["timestamp"].dt.hour
df_bids_imp_with_clk["day_of_week"] = df_bids_imp_with_clk[
"timestamp"
].dt.dayofweek
# Create new columns for browser and operating system
df_bids_imp_with_clk["browser"] = df_bids_imp_with_clk["user_agent"].apply(
get_browser
)
df_bids_imp_with_clk["os"] = df_bids_imp_with_clk["user_agent"].apply(get_os)
# Create one-hot encoding
one_hot_cols = [
"browser",
"os",
"region_id",
"ad_exchange_id",
"slot_visibility",
"slot_format",
"slot_width",
"slot_height",
"creative_id",
]
df_bids_imp_with_clk = pd.get_dummies(
df_bids_imp_with_clk, columns=one_hot_cols, prefix=one_hot_cols
)
# Convert boolean hot-encoded columns to int
for col in df_bids_imp_with_clk.columns:
if df_bids_imp_with_clk[col].dtype == "bool":
df_bids_imp_with_clk[col] = df_bids_imp_with_clk[col].astype(int)
missing_features = [
f for f in features if f not in df_bids_imp_with_clk.columns
]
for f in missing_features:
df_bids_imp_with_clk[f] = False
# Now, I only want to keep the columns that are in the features list plus the clicked column
df_bids_imp_with_clk = df_bids_imp_with_clk[
["clicked", "slot_floor_price", "bidding_price", "pay_price"] + features
]
# Save the dataset
# Convert boolean columns to integers (0/1)
bool_cols = df_bids_imp_with_clk.select_dtypes(include=["bool"]).columns
df_bids_imp_with_clk[bool_cols] = df_bids_imp_with_clk[bool_cols].astype(int)
df_bids_imp_with_clk.to_csv(f"bid_with_features_{date}.csv", index=False)
# Calculate the average CTR for all the days
average_ctrs[date] = df_bids_imp_with_clk["clicked"].mean()
# Store the average CTRs
import json
with open("average_ctrs.json", "w") as f:
json.dump(average_ctrs, f)Add Predicted CTR to the feature engineered dataset¶
The model created for each day is used by the DSP to predict the CTR of a bid. To simplify the simulation, and avoid uploading the model to the repository, we will run the model now and add the predicted CTR to the feature engineered dataset.
from Ax_utils import load_pctr_prediction_model
for date in [
"20130607",
"20130608",
"20130609",
"20130610",
"20130611",
"20130612",
]:
df_bids = pd.read_csv(f"dataset/bid_with_features_{date}.csv")
model, features, average_ctr, bid_to_pay_ratio = load_pctr_prediction_model(
date
)
df_bids["pctr"] = model.predict_proba(df_bids[features])[:, 1]
# Save the dataset
df_bids.to_csv(f"dataset/bid_with_features_and_pctr_{date}.csv", index=False)