Ablation Studies¶
Finally, the gold standard in building complex machine learning models is proving that each constituent part of the model contributes something to the proposed solution.
Ablation studies serve to dissect machine learning models and evaluate their impact.
In this section, we’ll finally discuss how to present complex machine learning models in publications and ensure the viability of each part we engineered to solve our particular problem set.
In [1]:
import pandas as pd
penguins = pd.read_csv('../data/penguins_clean.csv')
penguins.head()
Out[1]:
In [2]:
from sklearn.model_selection import train_test_split
num_features = ["Culmen Length (mm)", "Culmen Depth (mm)", "Flipper Length (mm)"]
cat_features = ["Sex"]
features = num_features + cat_features
target = ["Species"]
X_train, X_test, y_train, y_test = train_test_split(penguins[features], penguins[target[0]], stratify=penguins[target[0]], train_size=.7, random_state=42)
In [3]:
import numpy as np
from sklearn.svm import SVC
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import cross_val_score
num_transformer = StandardScaler()
cat_transformer = OneHotEncoder(handle_unknown='ignore')
preprocessor = ColumnTransformer(transformers=[
('num', num_transformer, num_features),
('cat', cat_transformer, cat_features)
])
model = Pipeline(steps=[
('preprocessor', preprocessor),
('classifier', SVC()),
])
model
Out[3]:
In [4]:
scores = cross_val_score(model, X_test, y_test, cv=10)
Let's compare this to a model that doesn't scale the numeric inputs.
In [5]:
scoring = pd.DataFrame(columns=["Average", "Deviation"])
scoring.loc["Full", :] = [scores.mean(), scores.std()]
scoring
Out[5]:
In [6]:
# num_transformer = StandardScaler()
cat_transformer = OneHotEncoder(handle_unknown='ignore')
preprocessor = ColumnTransformer(transformers=[
# ('num', num_transformer, num_features),
('cat', cat_transformer, cat_features)
])
model2 = Pipeline(steps=[
('preprocessor', preprocessor),
('classifier', SVC()),
])
scores = cross_val_score(model2, X_test, y_test, cv=10)
scoring.loc["No Standardisation",:] = [scores.mean(), scores.std()]
In [7]:
scoring
Out[7]:
In [8]:
num_transformer = StandardScaler()
cat_transformer = OneHotEncoder(handle_unknown='ignore', drop='if_binary')
preprocessor = ColumnTransformer(transformers=[
('num', num_transformer, num_features),
('cat', cat_transformer, cat_features)
])
model2 = Pipeline(steps=[
('preprocessor', preprocessor),
('classifier', SVC()),
])
scores = cross_val_score(model2, X_test, y_test, cv=10)
scoring.loc["Single Column Sex",:] = [scores.mean(), scores.std()]
scoring
Out[8]:
In [ ]: