Model Building Practical

A hands-on exercise for engineering features, tuning pipelines, and selecting a robust classification model with Python

Author

Complex Data Insights

Published

Aug 2026

COMPLEX DATA INSIGHTS
CDI Practicals

Hands-on exercises and step-by-step tools for doing, applying, and building evidence.

Model Building Practical

A hands-on exercise for engineering features, tuning pipelines, and selecting a robust classification model with Python

AuthorComplex Data Insights
PublishedAug 2026
ID CDI-DATASCIENCE-PRACTICAL-005
Type Practical Product
Audience Students, researchers, analysts, professionals, mentors, and learners developing practical data science skills
Theme Strong model building improves a defensible baseline through purposeful features, leakage-safe tuning, repeated validation, and honest model selection

How to Use This Practical

Complete this practical actively. Run each code block, inspect every result, and record what changed before moving on. The aim is not to obtain the largest score. The aim is to make model-development choices that can be explained, reproduced, and challenged.

You are encouraged to:

  • begin from the task and baseline established in Practical 004;
  • create features only when their meaning and availability are defensible;
  • keep every learned transformation inside a pipeline;
  • tune models using training data only;
  • separate model selection from final holdout evaluation;
  • examine discrimination, calibration, errors, and stability together;
  • prefer a simpler model when added complexity has no reliable benefit;
  • document rejected ideas as carefully as the selected model.

Follow the rendered HTML practical in a browser while working in VS Code. Use VS Code to run, adapt, and save the Python workflow. JupyterLab, Positron, or Google Colab can also be used, but repository paths and the final Quarto render are designed for VS Code.

Practical Overview

Practical 004 introduced the complete supervised-classification workflow. This practical concentrates on the next layer of judgement: improving a model without weakening the evidence supporting it.

You will revisit the fictional service-delivery question:

Can information available at the end of a service visit help identify records that may require follow-up?

You will compare a regularized logistic regression with a random forest. Both candidates will be evaluated as complete pipelines. You will add purposeful features, tune a limited set of hyperparameters, estimate selection performance with nested cross-validation, and diagnose the selected model.

The dataset is intentionally small. That makes instability visible and creates an important constraint: complex searches and confident claims are not justified.

Learning Outcomes

By completing this practical, you will be able to:

  • distinguish feature construction, feature transformation, and feature selection;
  • create domain-informed features without using the target;
  • prevent leakage during preprocessing and hyperparameter tuning;
  • design small, interpretable hyperparameter search spaces;
  • explain why the best tuning score is optimistically biased;
  • compare tuned pipelines with nested cross-validation;
  • summarize fold-to-fold performance variation;
  • inspect learning curves for data limitation and overfitting;
  • evaluate probability calibration and Brier score;
  • use permutation importance without claiming causality;
  • perform structured false-positive and false-negative analysis;
  • select a model using evidence, simplicity, and intended use;
  • save and verify a complete fitted pipeline.

What You Will Produce

By the end of the practical, you will create:

  • a model-development brief;
  • a feature-engineering plan and feature dictionary;
  • base and engineered feature sets;
  • tuned logistic-regression and random-forest pipelines;
  • nested cross-validation results;
  • a hyperparameter search summary;
  • a learning-curve figure;
  • a calibration figure and table;
  • a holdout performance table and confusion matrix;
  • a permutation-importance table and figure;
  • an error-analysis table;
  • a model-selection decision log;
  • a saved end-to-end pipeline;
  • a model-building report and portfolio statement.

Suggested output structure:

results/
├── figures/
│   ├── nested-cv-performance.png
│   ├── learning-curve.png
│   ├── calibration-curve.png
│   ├── holdout-confusion-matrix.png
│   └── permutation-importance.png
├── models/
│   └── selected-follow-up-pipeline.joblib
└── tables/
    ├── development-brief.csv
    ├── feature-plan.csv
    ├── feature-dictionary.csv
    ├── search-results.csv
    ├── nested-cv-results.csv
    ├── calibration-results.csv
    ├── holdout-performance.csv
    ├── holdout-predictions.csv
    ├── permutation-importance.csv
    ├── error-analysis.csv
    ├── model-selection-log.csv
    └── model-building-report.md

Tools and Requirements

You need Python 3.10 or later, VS Code with the Python and Jupyter extensions, Quarto, and these packages:

python -m pip install -r requirements.txt

The practical uses:

data/cleaned-service-data.csv

Keep the input unchanged. All derived tables, figures, and models belong under results/.

The supplied project files also include a standalone, script-first workflow. After installing the requirements, run:

bash scripts/bash/run-workflow.sh

This regenerates the deterministic synthetic dataset, executes the complete model-development workflow, verifies the fitted pipeline, and writes an audit manifest. The embedded code remains available for step-by-step learning; the standalone scripts provide a reproducible end-to-end check.

Project Structure

model-building-practical/
├── assets/css/
├── data/
│   └── cleaned-service-data.csv
├── results/
│   ├── figures/
│   ├── models/
│   ├── quality-control/
│   └── tables/
├── scripts/bash/
│   ├── build-product-zip.sh
│   ├── run-workflow.sh
│   └── split-practical.sh
├── scripts/python/
│   ├── generate-example-dataset.py
│   └── run-model-building.py
├── _quarto.yml
├── README.md
├── requirements.txt
└── practical.qmd

Open the repository root in VS Code so relative paths work consistently.

Relationship to Practical 004

Practical 004 established the task, cohort, baseline, initial feature set, candidate models, holdout principle, and responsible-use boundary. Practical 005 does not replace those foundations.

The progression is:

Practical 004 Practical 005
Compare simple untuned candidates Tune limited candidate pipelines
Use one cross-validation procedure Separate tuning from performance estimation
Inspect initial features Engineer and document new features
Evaluate discrimination and thresholds Add calibration, learning curves, and error analysis
Choose an introductory candidate Make a structured model-selection decision

If you have not completed Practical 004, read its sections on prediction framing, leakage, preprocessing pipelines, baselines, metrics, and holdout evaluation before continuing.

Dataset and Modelling Context

One row represents one fictional service record. The target is follow_up_required. Candidate information available at the end of a visit includes region, service type, age, visit date, and satisfaction score.

Direct identifiers, contact fields, the review flag, and the original target label are not predictors. Engineered features must be computable at the stated prediction moment. They must not summarize future records or use the target.

Learning exercise, not an operational model

Do not use this model to allocate or deny services, rank people, or automate follow-up. Operational use would require representative data, a validated target, domain and community review, privacy and impact assessment, external and prospective validation, monitoring, and a defined human decision process.

Define the Model-Development Brief

Freeze the rules before searching for a better model.

Component Decision for this practical
Unit One completed service visit
Positive class Follow-up required (Yes)
Prediction moment End of visit, after satisfaction is recorded
Primary selection metric ROC AUC
Supporting metrics Average precision, recall, precision, F1, Brier score
Holdout rule Isolate 20% before feature comparison or tuning
Candidate families Regularized logistic regression and random forest
Complexity rule Prefer the simpler model when performance is practically similar
Intended use Education and methodological review only
Show code
from pathlib import Path
import sys

import joblib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn

from sklearn.base import clone
from sklearn.calibration import CalibrationDisplay
from sklearn.compose import ColumnTransformer
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    ConfusionMatrixDisplay,
    accuracy_score,
    average_precision_score,
    brier_score_loss,
    f1_score,
    log_loss,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    GridSearchCV,
    StratifiedKFold,
    cross_validate,
    learning_curve,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
Show code
RANDOM_STATE = 42
DATA_PATH = Path("data/cleaned-service-data.csv")
RESULTS_DIR = Path("results")
FIGURES_DIR = RESULTS_DIR / "figures"
TABLES_DIR = RESULTS_DIR / "tables"
MODELS_DIR = RESULTS_DIR / "models"

for directory in [FIGURES_DIR, TABLES_DIR, MODELS_DIR]:
    directory.mkdir(parents=True, exist_ok=True)

plt.style.use("ggplot")
pd.set_option("display.max_columns", 40)

development_brief = pd.DataFrame({
    "component": [
        "unit", "positive_class", "prediction_moment", "primary_metric",
        "holdout_rule", "candidate_families", "complexity_rule", "intended_use"
    ],
    "decision": [
        "One completed service visit", "Follow-up required (Yes)",
        "End of visit after satisfaction is recorded", "ROC AUC",
        "Untouched 20% stratified holdout", "Logistic regression; random forest",
        "Prefer simpler model when performance is practically similar",
        "Education and methodological review only",
    ],
})
development_brief.to_csv(TABLES_DIR / "development-brief.csv", index=False)

print(f"Python: {sys.version.split()[0]}")
print(f"pandas: {pd.__version__}")
print(f"scikit-learn: {sklearn.__version__}")
development_brief
Python: 3.12.6
pandas: 2.3.3
scikit-learn: 1.9.0
component decision
0 unit One completed service visit
1 positive_class Follow-up required (Yes)
2 prediction_moment End of visit after satisfaction is recorded
3 primary_metric ROC AUC
4 holdout_rule Untouched 20% stratified holdout
5 candidate_families Logistic regression; random forest
6 complexity_rule Prefer simpler model when performance is pract...
7 intended_use Education and methodological review only

Load and Define the Cohort

Use the same cohort logic as Practical 004 so differences arise from model development rather than an unexplained population change.

Show code
data = pd.read_csv(DATA_PATH)

required_columns = {
    "record_id", "region", "service_type", "age", "visit_date",
    "satisfaction_score", "follow_up_required", "requires_review"
}
missing_columns = required_columns.difference(data.columns)
assert not missing_columns, f"Missing required columns: {sorted(missing_columns)}"

model_data = data.loc[data["requires_review"].eq(False)].copy()
model_data = model_data.loc[model_data["follow_up_required"].isin(["Yes", "No"])].copy()
model_data["visit_date"] = pd.to_datetime(model_data["visit_date"], errors="coerce")
model_data = model_data.loc[model_data["visit_date"].notna()].copy()
model_data["target"] = model_data["follow_up_required"].map({"No": 0, "Yes": 1}).astype(int)

duplicate_id_mask = model_data["record_id"].duplicated(keep=False)
model_data = model_data.loc[~duplicate_id_mask].copy()

assert model_data["record_id"].is_unique
assert model_data["target"].nunique() == 2

pd.Series({
    "source_rows": len(data),
    "modelling_rows": len(model_data),
    "positive_rows": int(model_data["target"].sum()),
    "positive_prevalence": model_data["target"].mean(),
})
source_rows            720.000000
modelling_rows         694.000000
positive_rows          275.000000
positive_prevalence      0.396254
dtype: float64

Freeze the Holdout Set

The holdout set is not a workspace for feature selection, tuning, threshold choice, or repeated comparison. It represents the final internal check after the specification has been selected.

Show code
development_data, holdout_data = train_test_split(
    model_data,
    test_size=0.20,
    stratify=model_data["target"],
    random_state=RANDOM_STATE,
)

assert set(development_data["record_id"]).isdisjoint(holdout_data["record_id"])

split_summary = pd.DataFrame({
    "split": ["development", "holdout"],
    "rows": [len(development_data), len(holdout_data)],
    "positives": [development_data["target"].sum(), holdout_data["target"].sum()],
    "prevalence": [development_data["target"].mean(), holdout_data["target"].mean()],
})
split_summary
split rows positives prevalence
0 development 555 220 0.396396
1 holdout 139 55 0.395683

Plan Feature Engineering

Feature engineering should express a plausible pattern more directly, improve measurement, or make a model easier to estimate. It should not be a search for accidental correlations.

For this exercise, create:

  • age_band: a coarse life-stage category;
  • visit_quarter: a broader seasonal representation than twelve months;
  • satisfaction_missing: whether satisfaction was unrecorded;
  • age_missing: whether age was unrecorded;
  • low_satisfaction: whether the recorded score is 1 or 2;
  • age_satisfaction: a numerical interaction between age and satisfaction.

The missingness indicators allow the model to distinguish an imputed value from a genuinely observed central value. Their usefulness remains an empirical question, and their meaning may reflect documentation practices rather than participant characteristics.

Show code
feature_plan = pd.DataFrame([
    ["age_band", "age", "Categorical", "Represent broad non-linear age patterns", "Available at prediction time"],
    ["visit_quarter", "visit_date", "Categorical", "Represent broad seasonality", "May encode operational change"],
    ["satisfaction_missing", "satisfaction_score", "Categorical", "Preserve information about missingness", "May encode documentation practice"],
    ["age_missing", "age", "Categorical", "Preserve information about missingness", "May encode documentation practice"],
    ["low_satisfaction", "satisfaction_score", "Categorical", "Represent a potentially useful threshold", "Threshold chosen before outcome review"],
    ["age_satisfaction", "age and satisfaction_score", "Numerical", "Represent a simple interaction", "Difficult to interpret in isolation"],
], columns=["feature", "source", "type", "rationale", "caution"])

feature_plan.to_csv(TABLES_DIR / "feature-plan.csv", index=False)
feature_plan
feature source type rationale caution
0 age_band age Categorical Represent broad non-linear age patterns Available at prediction time
1 visit_quarter visit_date Categorical Represent broad seasonality May encode operational change
2 satisfaction_missing satisfaction_score Categorical Preserve information about missingness May encode documentation practice
3 age_missing age Categorical Preserve information about missingness May encode documentation practice
4 low_satisfaction satisfaction_score Categorical Represent a potentially useful threshold Threshold chosen before outcome review
5 age_satisfaction age and satisfaction_score Numerical Represent a simple interaction Difficult to interpret in isolation

Construct Features Reproducibly

The function below uses only columns available at prediction time. It does not learn from the target or from population-level summaries. Learned operations—imputation, scaling, and category encoding—remain inside the candidate pipelines.

Show code
def construct_features(frame):
    features = pd.DataFrame(index=frame.index)
    features["region"] = frame["region"]
    features["service_type"] = frame["service_type"]
    features["age"] = pd.to_numeric(frame["age"], errors="coerce")
    features["satisfaction_score"] = pd.to_numeric(
        frame["satisfaction_score"], errors="coerce"
    )

    visit_date = pd.to_datetime(frame["visit_date"], errors="coerce")
    features["visit_month"] = visit_date.dt.month.astype("Int64").astype("string")
    features["visit_quarter"] = visit_date.dt.quarter.astype("Int64").astype("string")

    features["age_band"] = pd.cut(
        features["age"],
        bins=[0, 17, 34, 54, np.inf],
        labels=["Under 18", "18-34", "35-54", "55+"],
        include_lowest=True,
    ).astype("object")
    features["age_missing"] = np.where(features["age"].isna(), "Yes", "No")
    features["satisfaction_missing"] = np.where(
        features["satisfaction_score"].isna(), "Yes", "No"
    )
    features["low_satisfaction"] = np.select(
        [
            features["satisfaction_score"].isna(),
            features["satisfaction_score"].le(2),
        ],
        ["Unknown", "Yes"],
        default="No",
    )
    features["age_satisfaction"] = (
        features["age"] * features["satisfaction_score"]
    )
    return features


all_features = construct_features(model_data)
assert all_features.index.equals(model_data.index)
all_features.head()
region service_type age satisfaction_score visit_month visit_quarter age_band age_missing satisfaction_missing low_satisfaction age_satisfaction
0 South Screening 32.0 3.0 1 1 18-34 No No No 96.0
1 Central Consultation 55.0 2.0 5 2 55+ No No Yes 110.0
2 West Screening 18.0 3.0 7 3 18-34 No No No 54.0
3 East Outreach 60.0 5.0 3 1 55+ No No No 300.0
4 West Screening NaN 2.0 7 3 NaN Yes No Yes NaN

Create a Feature Dictionary

Show code
feature_dictionary = pd.DataFrame([
    ["region", "Categorical", "Source", "Service region"],
    ["service_type", "Categorical", "Source", "Type of service visit"],
    ["age", "Numerical", "Source", "Participant age in years"],
    ["satisfaction_score", "Numerical", "Source", "End-of-visit score"],
    ["visit_month", "Categorical", "Derived", "Calendar month of visit"],
    ["visit_quarter", "Categorical", "Derived", "Calendar quarter of visit"],
    ["age_band", "Categorical", "Derived", "Predefined broad age band"],
    ["age_missing", "Categorical", "Derived", "Age missingness indicator"],
    ["satisfaction_missing", "Categorical", "Derived", "Satisfaction missingness indicator"],
    ["low_satisfaction", "Categorical", "Derived", "Score of 1 or 2; Unknown if missing"],
    ["age_satisfaction", "Numerical", "Derived", "Age multiplied by satisfaction score"],
], columns=["feature", "type", "origin", "definition"])

feature_dictionary.to_csv(TABLES_DIR / "feature-dictionary.csv", index=False)
feature_dictionary
feature type origin definition
0 region Categorical Source Service region
1 service_type Categorical Source Type of service visit
2 age Numerical Source Participant age in years
3 satisfaction_score Numerical Source End-of-visit score
4 visit_month Categorical Derived Calendar month of visit
5 visit_quarter Categorical Derived Calendar quarter of visit
6 age_band Categorical Derived Predefined broad age band
7 age_missing Categorical Derived Age missingness indicator
8 satisfaction_missing Categorical Derived Satisfaction missingness indicator
9 low_satisfaction Categorical Derived Score of 1 or 2; Unknown if missing
10 age_satisfaction Numerical Derived Age multiplied by satisfaction score

Define Base and Engineered Feature Sets

The base set approximates the introductory model. The engineered set adds the planned representations. Comparing them tests whether the additional complexity improves development performance consistently.

Show code
base_numeric = ["age", "satisfaction_score"]
base_categorical = ["region", "service_type", "visit_month"]

engineered_numeric = base_numeric + ["age_satisfaction"]
engineered_categorical = base_categorical + [
    "visit_quarter", "age_band", "age_missing",
    "satisfaction_missing", "low_satisfaction",
]

X_development_all = construct_features(development_data)
X_holdout_all = construct_features(holdout_data)
y_development = development_data["target"].copy()
y_holdout = holdout_data["target"].copy()

assert not {"target", "follow_up_required", "record_id"}.intersection(X_development_all.columns)

Build Leakage-Safe Preprocessing

Imputation, scaling, and one-hot encoding are learned from the training partition within each resample. handle_unknown="ignore" allows a category seen later to be represented without breaking prediction.

Show code
def make_preprocessor(numeric_features, categorical_features):
    numeric_pipeline = Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
    ])
    categorical_pipeline = Pipeline([
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("encoder", OneHotEncoder(handle_unknown="ignore")),
    ])
    return ColumnTransformer([
        ("numeric", numeric_pipeline, numeric_features),
        ("categorical", categorical_pipeline, categorical_features),
    ])

Define Candidate Pipelines

Logistic regression offers a regularized linear probability model. The random forest can represent interactions and non-linearities, but has more flexibility and may be unstable in a small dataset.

Show code
logistic_pipeline = Pipeline([
    ("preprocess", make_preprocessor(engineered_numeric, engineered_categorical)),
    ("model", LogisticRegression(max_iter=5000, random_state=RANDOM_STATE)),
])

forest_pipeline = Pipeline([
    ("preprocess", make_preprocessor(engineered_numeric, engineered_categorical)),
    ("model", RandomForestClassifier(random_state=RANDOM_STATE, n_jobs=1)),
])

Design Purposeful Search Spaces

Large searches can overfit the resampling procedure, consume time, and create a false impression of precision. Use a small grid tied to understandable modelling choices.

Show code
search_spaces = {
    "logistic_regression": {
        "model__C": [0.01, 0.1, 1.0, 10.0],
        "model__class_weight": [None, "balanced"],
    },
    "random_forest": {
        "model__n_estimators": [200],
        "model__max_depth": [2, 4, None],
        "model__min_samples_leaf": [2, 5, 10],
        "model__max_features": ["sqrt", 0.75],
        "model__class_weight": [None, "balanced"],
    },
}

pd.Series({name: np.prod([len(v) for v in grid.values()]) for name, grid in search_spaces.items()}, name="combinations")
logistic_regression     8
random_forest          36
Name: combinations, dtype: int64

Why Nested Cross-Validation

Ordinary grid search reports the best average score among many tried settings. That score helped choose the setting, so it is not an independent estimate of the selected procedure.

Nested cross-validation separates two jobs:

  1. the inner loop selects hyperparameters;
  2. the outer loop estimates how the complete selection procedure performs on unseen development folds.

The untouched holdout set remains outside both loops.

Run Nested Cross-Validation

Use identical outer splits for both model families. With a small dataset, report the individual outer-fold scores and their spread—not only their mean.

Show code
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)
inner_cv = StratifiedKFold(n_splits=4, shuffle=True, random_state=RANDOM_STATE + 1)

candidates = {
    "logistic_regression": logistic_pipeline,
    "random_forest": forest_pipeline,
}

nested_rows = []
best_parameter_rows = []

for model_name, pipeline in candidates.items():
    for fold, (train_index, validation_index) in enumerate(
        outer_cv.split(X_development_all, y_development), start=1
    ):
        X_inner_train = X_development_all.iloc[train_index]
        X_outer_validation = X_development_all.iloc[validation_index]
        y_inner_train = y_development.iloc[train_index]
        y_outer_validation = y_development.iloc[validation_index]

        search = GridSearchCV(
            estimator=clone(pipeline),
            param_grid=search_spaces[model_name],
            scoring="roc_auc",
            cv=inner_cv,
            n_jobs=-1,
            refit=True,
        )
        search.fit(X_inner_train, y_inner_train)
        probability = search.predict_proba(X_outer_validation)[:, 1]
        prediction = (probability >= 0.50).astype(int)

        nested_rows.append({
            "model": model_name,
            "outer_fold": fold,
            "roc_auc": roc_auc_score(y_outer_validation, probability),
            "average_precision": average_precision_score(y_outer_validation, probability),
            "brier_score": brier_score_loss(y_outer_validation, probability),
            "recall": recall_score(y_outer_validation, prediction, zero_division=0),
            "precision": precision_score(y_outer_validation, prediction, zero_division=0),
            "inner_best_roc_auc": search.best_score_,
        })
        best_parameter_rows.append({
            "model": model_name,
            "outer_fold": fold,
            **search.best_params_,
        })

nested_results = pd.DataFrame(nested_rows)
nested_results.to_csv(TABLES_DIR / "nested-cv-results.csv", index=False)
nested_results
model outer_fold roc_auc average_precision brier_score recall precision inner_best_roc_auc
0 logistic_regression 1 0.625509 0.530503 0.240781 0.522727 0.469388 0.696913
1 logistic_regression 2 0.662822 0.561830 0.230720 0.659091 0.527273 0.689281
2 logistic_regression 3 0.747626 0.681736 0.198725 0.522727 0.696970 0.639586
3 logistic_regression 4 0.738467 0.681266 0.208746 0.568182 0.581395 0.648915
4 logistic_regression 5 0.619403 0.521254 0.235357 0.386364 0.629630 0.682412
5 random_forest 1 0.592944 0.514823 0.247542 0.431818 0.475000 0.709634
6 random_forest 2 0.714043 0.658949 0.219690 0.227273 0.769231 0.695726
7 random_forest 3 0.700136 0.567858 0.213278 0.568182 0.675676 0.665282
8 random_forest 4 0.787992 0.734613 0.224062 0.454545 0.689655 0.645183
9 random_forest 5 0.604817 0.481151 0.235518 0.250000 0.550000 0.691062

Summarize Nested Performance

Show code
nested_summary = (
    nested_results.groupby("model")
    .agg(
        mean_roc_auc=("roc_auc", "mean"),
        sd_roc_auc=("roc_auc", "std"),
        min_roc_auc=("roc_auc", "min"),
        max_roc_auc=("roc_auc", "max"),
        mean_average_precision=("average_precision", "mean"),
        mean_brier_score=("brier_score", "mean"),
        mean_recall=("recall", "mean"),
        mean_precision=("precision", "mean"),
    )
    .reset_index()
    .sort_values("mean_roc_auc", ascending=False)
)
nested_summary
model mean_roc_auc sd_roc_auc min_roc_auc max_roc_auc mean_average_precision mean_brier_score mean_recall mean_precision
1 random_forest 0.679986 0.081329 0.592944 0.787992 0.591479 0.228018 0.386364 0.631912
0 logistic_regression 0.678765 0.061074 0.619403 0.747626 0.595318 0.222866 0.531818 0.580931
Show code
fig, ax = plt.subplots(figsize=(8, 5))
models_in_order = nested_summary["model"].tolist()
positions = {name: index for index, name in enumerate(models_in_order)}

for model_name, group in nested_results.groupby("model"):
    x = np.repeat(positions[model_name], len(group))
    ax.scatter(x, group["roc_auc"], s=55, alpha=0.8, label=model_name)
    ax.hlines(
        group["roc_auc"].mean(), positions[model_name] - 0.22,
        positions[model_name] + 0.22, colors="black", linewidth=2
    )

ax.set_xticks(range(len(models_in_order)), [name.replace("_", " ").title() for name in models_in_order])
ax.set_ylabel("Outer-fold ROC AUC")
ax.set_ylim(0, 1)
ax.set_title("Nested cross-validation performance")
fig.tight_layout()
fig.savefig(FIGURES_DIR / "nested-cv-performance.png", dpi=300, bbox_inches="tight")
plt.show()

Outer-fold ROC AUC for each tuned model-building procedure.

Look for overlap, weak folds, and large variation. A small difference in mean ROC AUC is not automatically a meaningful advantage.

Inspect Hyperparameter Stability

If different outer folds select very different settings, the apparent optimum may be unstable.

Show code
best_parameters = pd.DataFrame(best_parameter_rows)
best_parameters
model outer_fold model__C model__class_weight model__max_depth model__max_features model__min_samples_leaf model__n_estimators
0 logistic_regression 1 0.1 balanced NaN NaN NaN NaN
1 logistic_regression 2 0.1 balanced NaN NaN NaN NaN
2 logistic_regression 3 0.1 None NaN NaN NaN NaN
3 logistic_regression 4 0.1 balanced NaN NaN NaN NaN
4 logistic_regression 5 0.1 None NaN NaN NaN NaN
5 random_forest 1 NaN balanced 2.0 0.75 10.0 200.0
6 random_forest 2 NaN None 2.0 sqrt 2.0 200.0
7 random_forest 3 NaN None NaN 0.75 10.0 200.0
8 random_forest 4 NaN balanced 2.0 sqrt 10.0 200.0
9 random_forest 5 NaN None 4.0 sqrt 10.0 200.0

Fit Final Development Searches

After estimating the selection procedure, fit each search to the complete development set. These searches define the candidate specifications to compare before the holdout is opened.

Show code
final_searches = {}
search_result_frames = []

for model_name, pipeline in candidates.items():
    search = GridSearchCV(
        estimator=clone(pipeline),
        param_grid=search_spaces[model_name],
        scoring="roc_auc",
        cv=inner_cv,
        n_jobs=-1,
        refit=True,
        return_train_score=True,
    )
    search.fit(X_development_all, y_development)
    final_searches[model_name] = search

    results_frame = pd.DataFrame(search.cv_results_)
    results_frame.insert(0, "model", model_name)
    search_result_frames.append(results_frame)

search_results = pd.concat(search_result_frames, ignore_index=True)
search_results.to_csv(TABLES_DIR / "search-results.csv", index=False)

for name, search in final_searches.items():
    print(name, search.best_params_, round(search.best_score_, 3))
logistic_regression {'model__C': 0.1, 'model__class_weight': 'balanced'} 0.681
random_forest {'model__class_weight': None, 'model__max_depth': 2, 'model__max_features': 'sqrt', 'model__min_samples_leaf': 10, 'model__n_estimators': 200} 0.685

Apply the Model-Selection Rule

Use a tolerance rule to avoid choosing complexity based on a negligible estimated difference. Here, the random forest must exceed logistic regression by at least 0.02 mean outer-fold ROC AUC. Otherwise, select logistic regression.

Show code
mean_auc = nested_summary.set_index("model")["mean_roc_auc"]
complexity_margin = 0.02

if mean_auc["random_forest"] >= mean_auc["logistic_regression"] + complexity_margin:
    selected_model_name = "random_forest"
    selection_reason = "Random forest exceeded the predeclared complexity margin."
else:
    selected_model_name = "logistic_regression"
    selection_reason = "Random forest did not exceed the predeclared complexity margin; prefer logistic regression."

selected_pipeline = final_searches[selected_model_name].best_estimator_
print(selected_model_name)
print(selection_reason)
logistic_regression
Random forest did not exceed the predeclared complexity margin; prefer logistic regression.

This rule is instructional, not universal. In real work, define practical equivalence using consequences, uncertainty, compute cost, interpretability, and maintenance requirements.

Inspect the Learning Curve

A learning curve compares training and validation scores as the training sample grows. A persistent large gap may indicate overfitting. Two low curves may indicate weak signal, poor features, or insufficient information.

Show code
learning_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE + 2)
train_sizes, train_scores, validation_scores = learning_curve(
    clone(selected_pipeline),
    X_development_all,
    y_development,
    cv=learning_cv,
    scoring="roc_auc",
    train_sizes=np.linspace(0.40, 1.0, 4),
    n_jobs=-1,
)

learning_results = pd.DataFrame({
    "training_rows": train_sizes,
    "training_mean": train_scores.mean(axis=1),
    "training_sd": train_scores.std(axis=1),
    "validation_mean": validation_scores.mean(axis=1),
    "validation_sd": validation_scores.std(axis=1),
})
learning_results
training_rows training_mean training_sd validation_mean validation_sd
0 177 0.788980 0.025306 0.667368 0.071106
1 266 0.759506 0.019050 0.668046 0.083932
2 355 0.755738 0.020288 0.688263 0.067696
3 444 0.744199 0.014600 0.675645 0.063609
Show code
fig, ax = plt.subplots(figsize=(8, 5))
for prefix, label, color in [
    ("training", "Training", "#1b6f5a"),
    ("validation", "Cross-validation", "#b35c1e"),
]:
    mean = learning_results[f"{prefix}_mean"]
    sd = learning_results[f"{prefix}_sd"]
    ax.plot(learning_results["training_rows"], mean, marker="o", label=label, color=color)
    ax.fill_between(learning_results["training_rows"], mean - sd, mean + sd, alpha=0.18, color=color)

ax.set_xlabel("Training rows")
ax.set_ylabel("ROC AUC")
ax.set_ylim(0, 1)
ax.set_title("Learning curve for selected specification")
ax.legend()
fig.tight_layout()
fig.savefig(FIGURES_DIR / "learning-curve.png", dpi=300, bbox_inches="tight")
plt.show()

Learning curve for the selected model specification using development data only.

Examine Development Calibration

Discrimination asks whether higher-risk records tend to receive higher probabilities. Calibration asks whether predicted probabilities correspond to observed frequencies.

Generate out-of-fold predictions from the selected specification so each development record is predicted by a model that did not train on that record.

Show code
from sklearn.model_selection import cross_val_predict

oof_probability = cross_val_predict(
    clone(selected_pipeline),
    X_development_all,
    y_development,
    cv=learning_cv,
    method="predict_proba",
    n_jobs=-1,
)[:, 1]

development_calibration = pd.DataFrame({
    "metric": ["ROC AUC", "Average precision", "Brier score", "Log loss"],
    "value": [
        roc_auc_score(y_development, oof_probability),
        average_precision_score(y_development, oof_probability),
        brier_score_loss(y_development, oof_probability),
        log_loss(y_development, oof_probability),
    ],
})
development_calibration.to_csv(TABLES_DIR / "calibration-results.csv", index=False)
development_calibration
metric value
0 ROC AUC 0.669132
1 Average precision 0.564381
2 Brier score 0.228148
3 Log loss 0.649669
Show code
fig, ax = plt.subplots(figsize=(7, 6))
CalibrationDisplay.from_predictions(
    y_development,
    oof_probability,
    n_bins=5,
    strategy="quantile",
    name=selected_model_name.replace("_", " ").title(),
    ax=ax,
)
ax.set_title("Development-set probability calibration")
fig.tight_layout()
fig.savefig(FIGURES_DIR / "calibration-curve.png", dpi=300, bbox_inches="tight")
plt.show()

Out-of-fold calibration curve for the selected development specification.

With few observations per bin, calibration points are unstable. Do not infer that a smooth-looking curve proves reliable probabilities.

Evaluate the Untouched Holdout Set

The model family, feature set, hyperparameters, and 0.50 classification threshold are now fixed. Evaluate the selected pipeline once.

Show code
holdout_probability = selected_pipeline.predict_proba(X_holdout_all)[:, 1]
holdout_prediction = (holdout_probability >= 0.50).astype(int)

holdout_performance = pd.DataFrame([{
    "model": selected_model_name,
    "threshold": 0.50,
    "rows": len(y_holdout),
    "positives": int(y_holdout.sum()),
    "accuracy": accuracy_score(y_holdout, holdout_prediction),
    "precision": precision_score(y_holdout, holdout_prediction, zero_division=0),
    "recall": recall_score(y_holdout, holdout_prediction, zero_division=0),
    "f1": f1_score(y_holdout, holdout_prediction, zero_division=0),
    "roc_auc": roc_auc_score(y_holdout, holdout_probability),
    "average_precision": average_precision_score(y_holdout, holdout_probability),
    "brier_score": brier_score_loss(y_holdout, holdout_probability),
    "log_loss": log_loss(y_holdout, holdout_probability),
}])
holdout_performance.to_csv(TABLES_DIR / "holdout-performance.csv", index=False)
holdout_performance.round(3)
model threshold rows positives accuracy precision recall f1 roc_auc average_precision brier_score log_loss
0 logistic_regression 0.5 139 55 0.626 0.526 0.545 0.536 0.649 0.592 0.228 0.649
Show code
holdout_predictions = pd.DataFrame({
    "record_id": holdout_data["record_id"].values,
    "observed": y_holdout.values,
    "predicted_probability": holdout_probability,
    "predicted_class": holdout_prediction,
})
holdout_predictions["error_type"] = np.select(
    [
        (holdout_predictions["observed"] == 1) & (holdout_predictions["predicted_class"] == 0),
        (holdout_predictions["observed"] == 0) & (holdout_predictions["predicted_class"] == 1),
    ],
    ["false_negative", "false_positive"],
    default="correct",
)
holdout_predictions.to_csv(TABLES_DIR / "holdout-predictions.csv", index=False)
holdout_predictions.sort_values("predicted_probability", ascending=False)
record_id observed predicted_probability predicted_class error_type
133 SRV-0488 1 0.828726 1 correct
85 SRV-0613 1 0.792230 1 correct
69 SRV-0551 1 0.777970 1 correct
137 SRV-0704 0 0.771789 1 false_positive
16 SRV-0238 1 0.758151 1 correct
... ... ... ... ... ...
52 SRV-0138 1 0.265177 0 false_negative
92 SRV-0452 0 0.255651 0 correct
34 SRV-0680 1 0.252489 0 false_negative
109 SRV-0080 0 0.232803 0 correct
4 SRV-0390 1 0.225980 0 false_negative

139 rows × 5 columns

Inspect the Holdout Confusion Matrix

Show code
fig, ax = plt.subplots(figsize=(6, 5))
ConfusionMatrixDisplay.from_predictions(
    y_holdout,
    holdout_prediction,
    display_labels=["No follow-up", "Follow-up"],
    cmap="Blues",
    colorbar=False,
    ax=ax,
)
ax.set_title("Selected model: holdout errors")
fig.tight_layout()
fig.savefig(FIGURES_DIR / "holdout-confusion-matrix.png", dpi=300, bbox_inches="tight")
plt.show()

Confusion matrix for the selected model on the untouched holdout set at threshold 0.50.

Report error counts alongside rates. With a small holdout, one changed prediction can substantially alter a metric.

Calculate Permutation Importance

Permutation importance measures how much holdout ROC AUC decreases when one input column is shuffled. It evaluates reliance on a feature in this fitted pipeline; it does not show causality, fairness, or the intrinsic value of that feature.

Show code
importance = permutation_importance(
    selected_pipeline,
    X_holdout_all,
    y_holdout,
    scoring="roc_auc",
    n_repeats=30,
    random_state=RANDOM_STATE,
    n_jobs=-1,
)

importance_table = pd.DataFrame({
    "feature": X_holdout_all.columns,
    "mean_importance": importance.importances_mean,
    "sd_importance": importance.importances_std,
}).sort_values("mean_importance", ascending=False)

importance_table.to_csv(TABLES_DIR / "permutation-importance.csv", index=False)
importance_table
feature mean_importance sd_importance
9 low_satisfaction 0.068405 0.023603
1 service_type 0.025180 0.028819
5 visit_quarter 0.015296 0.010436
10 age_satisfaction 0.013110 0.008183
3 satisfaction_score 0.010671 0.006926
6 age_band 0.010664 0.012179
4 visit_month 0.001991 0.007691
8 satisfaction_missing 0.000967 0.000365
7 age_missing -0.000108 0.000558
0 region -0.001219 0.017811
2 age -0.006955 0.019990
Show code
plot_importance = importance_table.sort_values("mean_importance").tail(12)
fig, ax = plt.subplots(figsize=(8, 6))
ax.barh(
    plot_importance["feature"],
    plot_importance["mean_importance"],
    xerr=plot_importance["sd_importance"],
    color="#2a7f62",
    alpha=0.85,
)
ax.axvline(0, color="black", linewidth=1)
ax.set_xlabel("Decrease in holdout ROC AUC after permutation")
ax.set_title("Permutation importance for selected pipeline")
fig.tight_layout()
fig.savefig(FIGURES_DIR / "permutation-importance.png", dpi=300, bbox_inches="tight")
plt.show()

Permutation importance on the holdout set; error bars show one standard deviation across repeats.

Correlated and redundant features can divide importance. Negative importance on a small holdout can occur through sampling variation.

Perform Structured Error Analysis

Error analysis asks where the model failed and what those failures suggest about data, measurement, and use. It must not become a disguised search that repeatedly adapts the model to the holdout set.

Show code
error_analysis = holdout_data[[
    "record_id", "region", "service_type", "age",
    "satisfaction_score", "visit_date", "follow_up_required"
]].copy()
error_analysis["predicted_probability"] = holdout_probability
error_analysis["predicted_class"] = holdout_prediction
error_analysis["error_type"] = holdout_predictions["error_type"].values
error_analysis["probability_distance_from_0_5"] = np.abs(
    error_analysis["predicted_probability"] - 0.50
)

error_analysis = error_analysis.sort_values(
    ["error_type", "probability_distance_from_0_5"],
    ascending=[True, False],
)
error_analysis.to_csv(TABLES_DIR / "error-analysis.csv", index=False)
error_analysis
record_id region service_type age satisfaction_score visit_date follow_up_required predicted_probability predicted_class error_type probability_distance_from_0_5
487 SRV-0488 South Screening 79.0 1.0 2025-02-09 Yes 0.828726 1 correct 0.328726
612 SRV-0613 East Referral 22.0 1.0 2025-07-21 Yes 0.792230 1 correct 0.292230
550 SRV-0551 Central Referral 60.0 4.0 2024-03-31 Yes 0.777970 1 correct 0.277970
79 SRV-0080 North Consultation 22.0 4.0 2025-12-11 No 0.232803 0 correct 0.267197
237 SRV-0238 Central Screening 70.0 1.0 2024-07-07 Yes 0.758151 1 correct 0.258151
... ... ... ... ... ... ... ... ... ... ... ...
316 SRV-0317 South Consultation 43.0 2.0 2024-09-15 No 0.521106 1 false_positive 0.021106
185 SRV-0186 East Outreach 40.0 3.0 2024-03-24 No 0.516927 1 false_positive 0.016927
210 SRV-0211 North Screening 62.0 3.0 2024-03-14 No 0.513848 1 false_positive 0.013848
228 SRV-0229 East Consultation 53.0 NaN 2025-04-24 No 0.506848 1 false_positive 0.006848
292 SRV-0293 West Consultation 45.0 NaN 2025-05-09 No 0.500622 1 false_positive 0.000622

139 rows × 11 columns

Review errors using these prompts:

  1. Are errors concentrated in a service type, region, or missingness pattern?
  2. Are wrong predictions confident or close to the threshold?
  3. Could the target itself be inconsistently documented?
  4. Is an omitted feature plausibly required at the prediction moment?
  5. Would the operational consequence differ for false positives and false negatives?
  6. Does any pattern need investigation in new data rather than another holdout-driven edit?

Do not publish record-level predictions or direct identifiers as portfolio evidence.

Compare Feature Sets as a Sensitivity Check

Evaluate whether engineered features improved the selected model family under cross-validation. This is a sensitivity check on development data, not a new reason to reopen holdout selection.

Show code
base_pipeline = Pipeline([
    ("preprocess", make_preprocessor(base_numeric, base_categorical)),
    ("model", clone(selected_pipeline.named_steps["model"])),
])

feature_set_rows = []
for feature_set_name, pipeline in {
    "base": base_pipeline,
    "engineered": clone(selected_pipeline),
}.items():
    scores = cross_validate(
        pipeline,
        X_development_all,
        y_development,
        cv=learning_cv,
        scoring={"roc_auc": "roc_auc", "average_precision": "average_precision"},
        n_jobs=-1,
    )
    feature_set_rows.append({
        "feature_set": feature_set_name,
        "mean_roc_auc": scores["test_roc_auc"].mean(),
        "sd_roc_auc": scores["test_roc_auc"].std(),
        "mean_average_precision": scores["test_average_precision"].mean(),
    })

feature_set_comparison = pd.DataFrame(feature_set_rows)
feature_set_comparison
feature_set mean_roc_auc sd_roc_auc mean_average_precision
0 base 0.660719 0.050373 0.577887
1 engineered 0.675645 0.063609 0.586155

If engineering adds no stable value, retaining the base representation may be the better decision.

Record the Model-Selection Decision

Show code
selected_nested = nested_summary.loc[
    nested_summary["model"].eq(selected_model_name)
].iloc[0]

model_selection_log = pd.DataFrame([{
    "selected_model": selected_model_name,
    "selected_hyperparameters": str(final_searches[selected_model_name].best_params_),
    "primary_metric": "ROC AUC",
    "mean_outer_cv_roc_auc": selected_nested["mean_roc_auc"],
    "sd_outer_cv_roc_auc": selected_nested["sd_roc_auc"],
    "complexity_margin": complexity_margin,
    "selection_reason": selection_reason,
    "holdout_opened_after_selection": True,
    "operational_use_approved": False,
}])
model_selection_log.to_csv(TABLES_DIR / "model-selection-log.csv", index=False)
model_selection_log.T
0
selected_model logistic_regression
selected_hyperparameters {'model__C': 0.1, 'model__class_weight': 'bala...
primary_metric ROC AUC
mean_outer_cv_roc_auc 0.678765
sd_outer_cv_roc_auc 0.061074
complexity_margin 0.02
selection_reason Random forest did not exceed the predeclared c...
holdout_opened_after_selection True
operational_use_approved False

Save and Verify the Complete Pipeline

The saved object must include preprocessing and the fitted model. Saving only the estimator would require later users to recreate transformations exactly.

Show code
MODEL_PATH = MODELS_DIR / "selected-follow-up-pipeline.joblib"
joblib.dump(selected_pipeline, MODEL_PATH)

reloaded_pipeline = joblib.load(MODEL_PATH)
reloaded_probability = reloaded_pipeline.predict_proba(X_holdout_all)[:, 1]

assert np.allclose(holdout_probability, reloaded_probability)
print(f"Verified: {MODEL_PATH}")
Verified: results/models/selected-follow-up-pipeline.joblib

Create the Model-Building Report

Show code
holdout_row = holdout_performance.iloc[0]

model_report = f"""# Model Building Report: Follow-up Classification



## Purpose
Educational demonstration of feature engineering, pipeline tuning, nested validation, and model selection.



## Development design
- Development records: {len(development_data)}
- Untouched holdout records: {len(holdout_data)}
- Primary selection metric: ROC AUC
- Inner validation: 4-fold stratified cross-validation
- Outer validation: 5-fold stratified cross-validation



## Selected specification
- Model: {selected_model_name}
- Hyperparameters: {final_searches[selected_model_name].best_params_}
- Selection rule: {selection_reason}
- Mean outer-fold ROC AUC: {selected_nested['mean_roc_auc']:.3f}
- Outer-fold ROC AUC SD: {selected_nested['sd_roc_auc']:.3f}



## Holdout results at threshold 0.50
- ROC AUC: {holdout_row['roc_auc']:.3f}
- Average precision: {holdout_row['average_precision']:.3f}
- Brier score: {holdout_row['brier_score']:.3f}
- Precision: {holdout_row['precision']:.3f}
- Recall: {holdout_row['recall']:.3f}
- F1: {holdout_row['f1']:.3f}



## Limitations
Small fictional dataset; uncertain target validity; internal random validation only; unstable fold and subgroup estimates; no external, temporal, or prospective validation; no impact assessment or monitoring.



## Permitted use
Learning, code review, and methodological discussion.



## Prohibited use
Allocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions.
"""

(TABLES_DIR / "model-building-report.md").write_text(model_report, encoding="utf-8")
print(model_report)
# Model Building Report: Follow-up Classification



## Purpose
Educational demonstration of feature engineering, pipeline tuning, nested validation, and model selection.



## Development design
- Development records: 555
- Untouched holdout records: 139
- Primary selection metric: ROC AUC
- Inner validation: 4-fold stratified cross-validation
- Outer validation: 5-fold stratified cross-validation



## Selected specification
- Model: logistic_regression
- Hyperparameters: {'model__C': 0.1, 'model__class_weight': 'balanced'}
- Selection rule: Random forest did not exceed the predeclared complexity margin; prefer logistic regression.
- Mean outer-fold ROC AUC: 0.679
- Outer-fold ROC AUC SD: 0.061



## Holdout results at threshold 0.50
- ROC AUC: 0.649
- Average precision: 0.592
- Brier score: 0.228
- Precision: 0.526
- Recall: 0.545
- F1: 0.536



## Limitations
Small fictional dataset; uncertain target validity; internal random validation only; unstable fold and subgroup estimates; no external, temporal, or prospective validation; no impact assessment or monitoring.



## Permitted use
Learning, code review, and methodological discussion.



## Prohibited use
Allocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions.

Interpretation and Limitations

Write a concise conclusion that answers four questions:

  1. Did the selected procedure perform consistently across outer folds?
  2. Did the more complex family clear the predeclared improvement margin?
  3. What did the untouched holdout show about discrimination, calibration, and errors?
  4. What evidence is still missing before any real-world use?

Important limitations include:

  • the data are fictional, small, and not representative of a defined population;
  • the target may reflect documentation or historical decisions rather than objective need;
  • derived features may encode operational practice rather than stable relationships;
  • nested cross-validation reduces selection bias but does not remove sampling uncertainty;
  • the random holdout does not test future periods, new sites, or changed workflows;
  • permutation importance is model-specific and non-causal;
  • error analysis on a small holdout is descriptive;
  • no model in this exercise is approved for deployment.

Export and Verify Results

Show code
expected_outputs = [
    TABLES_DIR / "development-brief.csv",
    TABLES_DIR / "feature-plan.csv",
    TABLES_DIR / "feature-dictionary.csv",
    TABLES_DIR / "search-results.csv",
    TABLES_DIR / "nested-cv-results.csv",
    TABLES_DIR / "calibration-results.csv",
    TABLES_DIR / "holdout-performance.csv",
    TABLES_DIR / "holdout-predictions.csv",
    TABLES_DIR / "permutation-importance.csv",
    TABLES_DIR / "error-analysis.csv",
    TABLES_DIR / "model-selection-log.csv",
    TABLES_DIR / "model-building-report.md",
    FIGURES_DIR / "nested-cv-performance.png",
    FIGURES_DIR / "learning-curve.png",
    FIGURES_DIR / "calibration-curve.png",
    FIGURES_DIR / "holdout-confusion-matrix.png",
    FIGURES_DIR / "permutation-importance.png",
    MODELS_DIR / "selected-follow-up-pipeline.joblib",
]

output_check = pd.DataFrame({
    "path": [str(path) for path in expected_outputs],
    "exists": [path.exists() for path in expected_outputs],
})
assert output_check["exists"].all(), "One or more expected outputs were not created."
output_check
path exists
0 results/tables/development-brief.csv True
1 results/tables/feature-plan.csv True
2 results/tables/feature-dictionary.csv True
3 results/tables/search-results.csv True
4 results/tables/nested-cv-results.csv True
5 results/tables/calibration-results.csv True
6 results/tables/holdout-performance.csv True
7 results/tables/holdout-predictions.csv True
8 results/tables/permutation-importance.csv True
9 results/tables/error-analysis.csv True
10 results/tables/model-selection-log.csv True
11 results/tables/model-building-report.md True
12 results/figures/nested-cv-performance.png True
13 results/figures/learning-curve.png True
14 results/figures/calibration-curve.png True
15 results/figures/holdout-confusion-matrix.png True
16 results/figures/permutation-importance.png True
17 results/models/selected-follow-up-pipeline.joblib True

Render from the VS Code terminal:

quarto render

Open products/index.html in a browser. Check navigation, code folding, tables, figures, warnings, and whether every written claim matches the displayed evidence.

Portfolio Evidence

Your portfolio evidence should show the reasoning path, not only the winning model:

  • the model-development brief and prediction moment;
  • the feature-engineering plan and cautions;
  • the leakage-safe pipeline design;
  • the restricted hyperparameter search spaces;
  • nested cross-validation results with variability;
  • the predeclared complexity rule;
  • learning-curve and calibration interpretation;
  • the single holdout evaluation;
  • permutation importance and error analysis with limitations;
  • the saved pipeline, selection log, and model-building report;
  • a clear non-deployment boundary.

Project Question

Use this portfolio framing:

Can purposeful feature engineering and leakage-safe model tuning improve follow-up classification beyond the introductory specification, and is any improvement stable enough to justify additional complexity?

For another dataset, revise the unit, outcome, prediction moment, primary metric, candidate families, validation design, and complexity rule.

Workflow Summary

Summarize the completed project as a reproducible sequence:

  1. restated the prediction task and intended-use boundary;
  2. reproduced the Practical 004 cohort rules;
  3. isolated a stratified holdout set;
  4. planned and documented target-independent features;
  5. constructed base and engineered feature sets;
  6. placed learned preprocessing inside pipelines;
  7. defined limited logistic-regression and random-forest search spaces;
  8. used inner folds for tuning and outer folds for estimation;
  9. applied a predeclared complexity margin;
  10. examined learning curves and out-of-fold calibration;
  11. evaluated the selected specification once on the holdout;
  12. inspected errors and permutation importance cautiously;
  13. saved, reloaded, and verified the complete pipeline;
  14. exported a model-selection log and report.

Quality Checks

Document that:

  • cohort rules match the prior practical;
  • direct identifiers, contact fields, review status, and target are excluded from predictors;
  • engineered features are available at the prediction moment;
  • feature construction does not use target values;
  • the holdout was isolated before model comparison;
  • imputation, scaling, and encoding occur inside pipelines;
  • both candidate families use identical outer folds;
  • the inner loop tunes and the outer loop estimates;
  • the primary metric and complexity rule were defined before holdout evaluation;
  • probabilities, not class labels, were used for ROC AUC, average precision, Brier score, and log loss;
  • holdout evaluation occurred once after specification selection;
  • importance is described as predictive reliance, not causation;
  • record-level outputs are not published;
  • the saved pipeline reproduces predictions;
  • the rendered HTML has been inspected.

Reflection and Next Steps

Write brief responses to these prompts:

  1. Which engineered feature had the clearest domain rationale?
  2. Which feature was most vulnerable to encoding documentation practice?
  3. How different were inner best scores from outer-fold scores?
  4. Were the selected hyperparameters stable across outer folds?
  5. Did the complex model clear the improvement margin?
  6. What did the learning curve suggest about more data?
  7. Were predicted probabilities adequately calibrated for the learning purpose?
  8. Which holdout errors appeared most consequential?
  9. Did permutation importance change how you understood the model?
  10. What should be validated temporally or externally next?

Practical 006 can continue from this saved pipeline into model deployment: prediction interfaces, input validation, packaging, tests, monitoring, and safe release decisions.

Completion Checklist

Before marking the practical complete, confirm that you have:

Practical Complete

You have completed Data Science Practical 005: Model Building Practical.

You now have a reproducible model-development workflow that moves beyond fitting candidate algorithms. It connects purposeful feature design, leakage-safe tuning, nested performance estimation, calibration, model diagnostics, and an explicit selection rule.

Keep the unchanged input data, complete practical.qmd, exported evidence, saved pipeline, decision log, report, and rendered HTML together. They form a traceable record of what was tried, what was selected, what remained uncertain, and why a fitted model is not yet a deployed product.

© Complex Data Insights | CDI Practicals

Hands-on exercises and step-by-step tools