Ablation Studies

Finally, the gold standard in building complex machine learning models is proving that each constituent part of the model contributes something to the proposed solution.

Ablation studies serve to dissect machine learning models and evaluate their impact.

In this section, we’ll finally discuss how to present complex machine learning models in publications and ensure the viability of each part we engineered to solve our particular problem set.

In [1]:
import pandas as pd
penguins = pd.read_csv('../data/penguins_clean.csv')
penguins.head()
Out[1]:
Culmen Length (mm) Culmen Depth (mm) Flipper Length (mm) Sex Species
0 39.1 18.7 181.0 MALE Adelie Penguin (Pygoscelis adeliae)
1 39.5 17.4 186.0 FEMALE Adelie Penguin (Pygoscelis adeliae)
2 40.3 18.0 195.0 FEMALE Adelie Penguin (Pygoscelis adeliae)
3 36.7 19.3 193.0 FEMALE Adelie Penguin (Pygoscelis adeliae)
4 39.3 20.6 190.0 MALE Adelie Penguin (Pygoscelis adeliae)
In [2]:
from sklearn.model_selection import train_test_split
num_features = ["Culmen Length (mm)", "Culmen Depth (mm)", "Flipper Length (mm)"]
cat_features = ["Sex"]
features = num_features + cat_features
target = ["Species"]

X_train, X_test, y_train, y_test = train_test_split(penguins[features], penguins[target[0]], stratify=penguins[target[0]], train_size=.7, random_state=42)
In [3]:
import numpy as np
from sklearn.svm import SVC
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import cross_val_score

num_transformer = StandardScaler()
cat_transformer = OneHotEncoder(handle_unknown='ignore')

preprocessor = ColumnTransformer(transformers=[
    ('num', num_transformer, num_features),
    ('cat', cat_transformer, cat_features)
])


model = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', SVC()),
])
model
Out[3]:
Pipeline(steps=[('preprocessor',
                 ColumnTransformer(transformers=[('num', StandardScaler(),
                                                  ['Culmen Length (mm)',
                                                   'Culmen Depth (mm)',
                                                   'Flipper Length (mm)']),
                                                 ('cat',
                                                  OneHotEncoder(handle_unknown='ignore'),
                                                  ['Sex'])])),
                ('classifier', SVC())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
In [4]:
scores = cross_val_score(model, X_test, y_test, cv=10)

Let's compare this to a model that doesn't scale the numeric inputs.

In [5]:
scoring = pd.DataFrame(columns=["Average", "Deviation"])
scoring.loc["Full", :] = [scores.mean(), scores.std()]
scoring
Out[5]:
Average Deviation
Full 1.0 0.0
In [6]:
# num_transformer = StandardScaler()
cat_transformer = OneHotEncoder(handle_unknown='ignore')

preprocessor = ColumnTransformer(transformers=[
    # ('num', num_transformer, num_features),
    ('cat', cat_transformer, cat_features)
])


model2 = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', SVC()),
])

scores = cross_val_score(model2, X_test, y_test, cv=10)

scoring.loc["No Standardisation",:] = [scores.mean(), scores.std()]
In [7]:
scoring
Out[7]:
Average Deviation
Full 1.0 0.0
No Standardisation 0.435455 0.045172
In [8]:
num_transformer = StandardScaler()
cat_transformer = OneHotEncoder(handle_unknown='ignore', drop='if_binary')

preprocessor = ColumnTransformer(transformers=[
    ('num', num_transformer, num_features),
    ('cat', cat_transformer, cat_features)
])


model2 = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', SVC()),
])

scores = cross_val_score(model2, X_test, y_test, cv=10)

scoring.loc["Single Column Sex",:] = [scores.mean(), scores.std()]
scoring
C:\tools\Anaconda3\envs\euroscipy-2022-ml-repro\lib\site-packages\sklearn\preprocessing\_encoders.py:188: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
  warnings.warn(
Out[8]:
Average Deviation
Full 1.0 0.0
No Standardisation 0.435455 0.045172
Single Column Sex 1.0 0.0
In [ ]: