Introduction to Machine Learning Practical

A hands-on exercise for building, evaluating, and communicating a responsible classification model with Python

Author

Complex Data Insights

Published

Aug 2026

COMPLEX DATA INSIGHTS
CDI Practicals

Hands-on exercises and step-by-step tools for doing, applying, and building evidence.

Introduction to Machine Learning Practical

A hands-on exercise for building, evaluating, and communicating a responsible classification model with Python

AuthorComplex Data Insights
PublishedAug 2026
ID CDI-DATASCIENCE-PRACTICAL-008
Type Practical Product
Audience Students, researchers, analysts, professionals, mentors, and learners developing practical data science skills
ThemeUseful machine learning begins with a clear prediction task, a defensible validation design, an honest baseline, and evidence-aware communication

How to Use This Practical

Complete this practical actively. Run every code block, inspect every result, and write your own interpretation before moving on. The example uses a small fictional service-delivery dataset so you can focus on the workflow rather than computing power.

You are encouraged to:

  • define the prediction moment and target before choosing a model;
  • separate training from testing before learning transformations;
  • compare every model with a simple baseline;
  • evaluate more than accuracy;
  • treat predicted probabilities as estimates, not facts;
  • check whether performance differs across meaningful groups;
  • document uncertainty, limitations, and intended use;
  • avoid deploying a model whose consequences have not been assessed.

Follow the rendered HTML practical in a browser while working in VS Code. Use VS Code to run, adapt, and save the Python workflow. JupyterLab, Positron, or Google Colab can also be used, but repository paths and the final Quarto render are designed for VS Code.

Practical Overview

Machine learning uses patterns in existing data to make predictions for new observations. This practical introduces supervised binary classification: each historical record has features and a known two-category outcome, and the model learns to estimate the probability of one category.

This practical begins the Machine Learning and Systems sequence and builds on the earlier Data Foundations and Databases and Pipelines practicals. You will use cleaned service-delivery records to explore a narrow operational question:

Can information available at the end of a service visit help identify records that may require follow-up?

You will compare a prevalence-only baseline, logistic regression, and a shallow decision tree. The goal is not to produce a deployable service-allocation system. It is to learn a reproducible modelling workflow and recognize when the evidence is too limited for operational use.

Learning Outcomes

By completing this practical, you will be able to:

  • distinguish prediction from explanation and causal inference;
  • define a target, features, observation unit, and prediction moment;
  • identify identifier, proxy, temporal, and preprocessing leakage;
  • create reproducible stratified training and test sets;
  • build mixed-type preprocessing inside a scikit-learn pipeline;
  • establish a dummy baseline;
  • train and compare logistic regression and decision-tree classifiers;
  • use cross-validation without touching the test set;
  • interpret accuracy, precision, recall, F1, ROC AUC, and confusion matrices;
  • examine predicted probabilities and decision thresholds;
  • perform a basic subgroup performance check;
  • save a fitted pipeline and portfolio-ready evidence;
  • communicate limitations and responsible-use boundaries.

What You Will Produce

By the end of the practical, you will create:

  • a machine-learning problem statement;
  • a feature and leakage review;
  • reproducible training and test sets;
  • a baseline and two candidate models;
  • cross-validation results;
  • a holdout performance table;
  • confusion-matrix, ROC, precision–recall, and threshold figures;
  • a subgroup performance table;
  • a coefficient table for interpretation;
  • a model card and decision log;
  • a saved end-to-end model pipeline;
  • a portfolio-ready project statement.

Suggested output structure:

results/
├── figures/
│   ├── target-balance.png
│   ├── confusion-matrices.png
│   ├── roc-curves.png
│   ├── precision-recall-curves.png
│   └── threshold-tradeoff.png
├── models/
│   └── logistic-follow-up-pipeline.joblib
└── tables/
    ├── feature-review.csv
    ├── cross-validation-results.csv
    ├── holdout-performance.csv
    ├── test-predictions.csv
    ├── threshold-results.csv
    ├── subgroup-performance.csv
    ├── logistic-coefficients.csv
    ├── model-decision-log.csv
    └── model-card.md

Tools and Requirements

You need Python 3.10 or later, VS Code with the Python and Jupyter extensions, Quarto, and these packages:

python -m pip install -r requirements.txt

The practical uses:

data/cleaned-service-data.csv

Keep the input unchanged. All derived tables, figures, and models belong under results/.

Project Structure

intro-to-machine-learning-practical/
├── assets/css/
├── data/
│   └── cleaned-service-data.csv
├── results/
│   ├── figures/
│   ├── models/
│   └── tables/
├── scripts/bash/
│   ├── build-product-zip.sh
│   ├── run-workflow.sh
│   └── split-practical.sh
├── scripts/python/
│   ├── generate-example-dataset.py
│   └── run-intro-machine-learning.py
├── _quarto.yml
└── practical.qmd

Open the repository root in VS Code so relative paths work consistently.

Dataset and Prediction Context

One row represents one fictional service record. Candidate predictors include region, service type, participant age, visit month, satisfaction score, and whether the participant was under 18. The binary target is follow_up_required.

The target balance and sample size will strongly affect evaluation. A model can appear accurate simply by predicting the common class. The records are fictional and do not represent a population, intervention, or validated operational process.

Direct identifiers—record ID, name, and email—are excluded. A direct identifier may let a flexible model memorize records without learning a generalizable relationship. Names and emails also create unnecessary privacy risk.

Learning exercise, not a deployment model

Do not use this example model to allocate care, deny services, rank people, or automate follow-up. Deployment would require representative data, an independently validated outcome, domain review, privacy and impact assessment, prospective validation, monitoring, and a clear human decision process.

Define the Prediction Task

Complete this specification before coding:

Component Example definition
Unit One completed service visit
Target Whether follow-up is recorded as required (Yes or No)
Positive class Yes
Prediction moment At the end of the visit, after satisfaction is recorded
Intended user Analyst evaluating whether a predictive workflow is feasible
Intended action No operational action; methodological learning only
More costly error A false negative may miss a record that needs review
Validation aim Estimate performance on unseen records from the same fictional process

The prediction moment matters. If prediction were required before the visit ended, satisfaction score would not yet be available and must be removed.

Prediction Is Not Explanation

A predictive association does not establish why follow-up is required. A model may exploit documentation habits, service availability, missingness, or historical decisions. Even strong test performance would not show that changing a predictor would change the outcome.

Use predictive language:

  • “the model assigned a higher estimated probability”;
  • “this feature was associated with predictions in this dataset”;
  • “holdout performance suggests limited or useful discrimination.”

Avoid causal language such as “age causes follow-up” or “the model proves which service creates need.”

Prepare the Machine-Learning Environment

Show code
from pathlib import Path
import sys

import joblib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn

from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    ConfusionMatrixDisplay,
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
    make_scorer,
)
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeClassifier
Show code
RANDOM_STATE = 42

DATA_PATH = Path("data/cleaned-service-data.csv")
RESULTS_DIR = Path("results")
FIGURES_DIR = RESULTS_DIR / "figures"
TABLES_DIR = RESULTS_DIR / "tables"
MODELS_DIR = RESULTS_DIR / "models"

for directory in [FIGURES_DIR, TABLES_DIR, MODELS_DIR]:
    directory.mkdir(parents=True, exist_ok=True)

pd.set_option("display.max_columns", 30)
plt.style.use("ggplot")

print(f"Python: {sys.version.split()[0]}")
print(f"pandas: {pd.__version__}")
print(f"scikit-learn: {sklearn.__version__}")
Python: 3.12.6
pandas: 2.3.3
scikit-learn: 1.9.0

Load and Inspect the Data

Show code
data = pd.read_csv(DATA_PATH)

required_columns = {
    "record_id", "region", "service_type", "age", "visit_date",
    "satisfaction_score", "follow_up_required", "requires_review"
}
missing_columns = required_columns.difference(data.columns)
assert not missing_columns, f"Missing required columns: {sorted(missing_columns)}"

print(f"Rows: {len(data):,}")
print(f"Columns: {data.shape[1]}")
data.head()
Rows: 360
Columns: 10
record_id participant_name contact_email region service_type age visit_date satisfaction_score follow_up_required requires_review
0 SRV-0001 Participant 0001 participant0001@example.org West Advisory NaN 2025-08-25 4.0 No False
1 SRV-0002 Participant 0002 participant0002@example.org East Assessment 52.0 2025-11-02 3.0 No False
2 SRV-0003 Participant 0003 participant0003@example.org Central Advisory 71.0 2025-11-16 3.0 Yes True
3 SRV-0004 Participant 0004 participant0004@example.org South Assessment 41.0 2025-12-07 4.0 No False
4 SRV-0005 Participant 0005 participant0005@example.org West Referral 55.0 2025-12-07 2.0 Yes False
Show code
pd.crosstab(
    data["follow_up_required"].fillna("Missing"),
    data["requires_review"].fillna("Missing"),
    margins=True,
)
requires_review False True All
follow_up_required
No 272 7 279
Yes 75 6 81
All 347 13 360

Records retained for review may contain unresolved dates, categories, or values. Excluding them defines the modelling population more clearly; it does not make the remaining sample representative. The source also contains reused record_id values attached to different people or visits. Because an ambiguous identifier can compromise row-level traceability and grouped validation, those duplicate-ID rows are excluded from this learning cohort and counted in the cohort summary.

Define the Modelling Cohort

Show code
model_data = data.loc[data["requires_review"].eq(False)].copy()
model_data = model_data.loc[model_data["follow_up_required"].isin(["Yes", "No"])].copy()
model_data["visit_date"] = pd.to_datetime(model_data["visit_date"], errors="coerce")
model_data = model_data.loc[model_data["visit_date"].notna()].copy()

model_data["target"] = model_data["follow_up_required"].map({"No": 0, "Yes": 1}).astype(int)
model_data["visit_month"] = model_data["visit_date"].dt.month.astype(str)
model_data["under_18"] = np.where(
    model_data["age"].isna(), "Unknown", np.where(model_data["age"] < 18, "Yes", "No")
)

# A record identifier should identify one observation.  The source file contains
# two identifiers that are reused for different people/visits, so retain neither
# ambiguous row in the modelling cohort and record the exclusion explicitly.
duplicate_id_mask = model_data["record_id"].duplicated(keep=False)
duplicate_id_rows = int(duplicate_id_mask.sum())
duplicate_id_values = sorted(model_data.loc[duplicate_id_mask, "record_id"].unique())
model_data = model_data.loc[~duplicate_id_mask].copy()

assert model_data["record_id"].is_unique, "Record IDs must be unique after exclusions."
assert model_data["target"].nunique() == 2, "Both target classes are required."

cohort_summary = pd.Series({
    "source_rows": len(data),
    "duplicate_id_rows_excluded": duplicate_id_rows,
    "modelling_rows": len(model_data),
    "positive_rows": int(model_data["target"].sum()),
    "positive_prevalence": model_data["target"].mean(),
})
cohort_summary
source_rows                   360.000000
duplicate_id_rows_excluded      4.000000
modelling_rows                343.000000
positive_rows                  75.000000
positive_prevalence             0.218659
dtype: float64

Examine Target Balance

Show code
target_counts = model_data["follow_up_required"].value_counts().reindex(["No", "Yes"])

fig, ax = plt.subplots(figsize=(6.5, 4.2))
ax.bar(target_counts.index, target_counts.values, color="#036281")
ax.set(title="Follow-up requirement in the modelling cohort", xlabel="Follow-up required", ylabel="Records")
for index, value in enumerate(target_counts.values):
    ax.text(index, value + 0.5, str(value), ha="center")
fig.tight_layout()
fig.savefig(FIGURES_DIR / "target-balance.png", dpi=200, bbox_inches="tight")
plt.show()

Follow-up target balance in the modelling cohort.

Class imbalance changes what “good” performance means. Record the positive prevalence; it supplies context for the dummy baseline and precision–recall curve.

Review Features and Leakage

Show code
feature_review = pd.DataFrame([
    ["age", "numeric", "Include", "Available at prediction time; may be missing"],
    ["satisfaction_score", "numeric", "Include conditionally", "Only valid if prediction occurs after scoring"],
    ["region", "categorical", "Include", "May encode structural or access differences; check subgroups"],
    ["service_type", "categorical", "Include", "Available at the end of the visit"],
    ["visit_month", "categorical", "Include", "Derived from visit date; captures seasonality, not identity"],
    ["under_18", "categorical", "Include", "Derived from age for an explicit age-group check"],
    ["record_id", "identifier", "Exclude", "Identifier; duplicate IDs are excluded from the modelling cohort and identifiers are never used as features"],
    ["participant_name", "identifier", "Exclude", "Privacy and memorization risk"],
    ["contact_email", "identifier", "Exclude", "Privacy and memorization risk"],
    ["follow_up_required", "target", "Exclude from features", "Direct target leakage"],
    ["requires_review", "post-cleaning flag", "Exclude", "Used to define cohort; may encode data-quality process"],
])

feature_review.columns = ["variable", "role", "decision", "reason"]
feature_review.to_csv(TABLES_DIR / "feature-review.csv", index=False)
feature_review
variable role decision reason
0 age numeric Include Available at prediction time; may be missing
1 satisfaction_score numeric Include conditionally Only valid if prediction occurs after scoring
2 region categorical Include May encode structural or access differences; c...
3 service_type categorical Include Available at the end of the visit
4 visit_month categorical Include Derived from visit date; captures seasonality,...
5 under_18 categorical Include Derived from age for an explicit age-group check
6 record_id identifier Exclude Identifier; duplicate IDs are excluded from th...
7 participant_name identifier Exclude Privacy and memorization risk
8 contact_email identifier Exclude Privacy and memorization risk
9 follow_up_required target Exclude from features Direct target leakage
10 requires_review post-cleaning flag Exclude Used to define cohort; may encode data-quality...

Leakage occurs when training uses information that would not be legitimately available for a new prediction. It can enter through target-derived columns, future information, preprocessing performed before splitting, repeated people across splits, or an unrealistic random split for time-dependent data.

Select Features and Target

Show code
numeric_features = ["age", "satisfaction_score"]
categorical_features = ["region", "service_type", "visit_month", "under_18"]
feature_columns = numeric_features + categorical_features

X = model_data[feature_columns].copy()
y = model_data["target"].copy()

assert not set(feature_columns).intersection({
    "record_id", "participant_name", "contact_email",
    "follow_up_required", "requires_review", "target"
})

X.head()
age satisfaction_score region service_type visit_month under_18
0 NaN 4.0 West Advisory 8 Unknown
1 52.0 3.0 East Assessment 11 No
3 41.0 4.0 South Assessment 12 No
4 55.0 2.0 West Referral 12 No
5 33.0 5.0 Central Assessment 4 No

Create Training and Test Sets

The holdout test set is used once for the final comparison. Stratification approximately preserves class balance in both subsets.

Show code
row_indices = np.arange(len(model_data))
train_indices, test_indices = train_test_split(
    row_indices,
    test_size=0.25,
    random_state=RANDOM_STATE,
    stratify=y,
)

X_train = X.iloc[train_indices].copy()
X_test = X.iloc[test_indices].copy()
y_train = y.iloc[train_indices].copy()
y_test = y.iloc[test_indices].copy()

split_summary = pd.DataFrame({
    "split": ["training", "test"],
    "rows": [len(y_train), len(y_test)],
    "positive_rate": [y_train.mean(), y_test.mean()],
})
split_summary
split rows positive_rate
0 training 257 0.217899
1 test 86 0.220930
When a random split is not valid

Use group-aware splitting when one person contributes multiple records. Use a temporal split when the real task predicts future periods from earlier periods. A random split can overstate performance when related or future observations appear in both sets.

Build Preprocessing Pipelines

Numeric values are median-imputed and standardized. Categorical values are most-frequent-imputed and one-hot encoded. All transformations live inside the pipeline, so they are learned from training folds only.

Show code
numeric_pipeline = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

categorical_pipeline = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer(transformers=[
    ("numeric", numeric_pipeline, numeric_features),
    ("categorical", categorical_pipeline, categorical_features),
])

Do not impute, scale, or select categories using the complete dataset before splitting. Those apparently small actions allow test information to influence training.

Establish a Baseline

The dummy model predicts according to the training prevalence and uses no features. A candidate model that cannot improve meaningfully on this baseline has not demonstrated useful signal.

Show code
models = {
    "Dummy baseline": Pipeline(steps=[
        ("preprocessor", preprocessor),
        ("classifier", DummyClassifier(strategy="prior")),
    ]),
    "Logistic regression": Pipeline(steps=[
        ("preprocessor", preprocessor),
        ("classifier", LogisticRegression(max_iter=2000, class_weight="balanced", random_state=RANDOM_STATE)),
    ]),
    "Decision tree": Pipeline(steps=[
        ("preprocessor", preprocessor),
        ("classifier", DecisionTreeClassifier(
            max_depth=3, min_samples_leaf=8, class_weight="balanced", random_state=RANDOM_STATE
        )),
    ]),
}

Choose Evaluation Metrics

  • Accuracy is the proportion of all predictions that are correct.
  • Precision asks: among predicted positives, how many are positive?
  • Recall asks: among actual positives, how many are found?
  • F1 balances precision and recall through their harmonic mean.
  • ROC AUC measures ranking discrimination across thresholds.
  • Average precision summarizes the precision–recall curve and is especially informative when the positive class is uncommon.

No metric determines usefulness by itself. Choose metrics from the decision context and error costs before comparing models.

Compare Models with Cross-Validation

Five-fold stratified cross-validation repeatedly trains on four folds and validates on the remaining fold. The test set remains untouched.

Show code
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)
scoring = {
    "accuracy": "accuracy",
    "precision": make_scorer(precision_score, zero_division=0),
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}

cv_rows = []
for model_name, pipeline in models.items():
    scores = cross_validate(pipeline, X_train, y_train, cv=cv, scoring=scoring)
    for fold in range(cv.get_n_splits()):
        row = {"model": model_name, "fold": fold + 1}
        row.update({metric: scores[f"test_{metric}"][fold] for metric in scoring})
        cv_rows.append(row)

cv_results = pd.DataFrame(cv_rows)
cv_results.to_csv(TABLES_DIR / "cross-validation-results.csv", index=False)

cv_summary = (
    cv_results.groupby("model")[list(scoring)]
    .agg(["mean", "std"])
    .round(3)
)
cv_summary
accuracy precision recall f1 roc_auc average_precision
mean std mean std mean std mean std mean std mean std
model
Decision tree 0.517 0.113 0.175 0.062 0.397 0.315 0.231 0.116 0.495 0.066 0.228 0.035
Dummy baseline 0.782 0.007 0.000 0.000 0.000 0.000 0.000 0.000 0.500 0.000 0.218 0.007
Logistic regression 0.561 0.070 0.238 0.068 0.468 0.157 0.315 0.094 0.546 0.106 0.288 0.067

Variation across folds is evidence, not noise to hide. With a small dataset, a modest difference between candidate models may not be stable.

Fit Models and Evaluate the Holdout Set

Show code
fitted_models = {}
prediction_rows = []
performance_rows = []

for model_name, pipeline in models.items():
    fitted = pipeline.fit(X_train, y_train)
    fitted_models[model_name] = fitted
    predicted_class = fitted.predict(X_test)
    predicted_probability = fitted.predict_proba(X_test)[:, 1]

    performance_rows.append({
        "model": model_name,
        "accuracy": accuracy_score(y_test, predicted_class),
        "precision": precision_score(y_test, predicted_class, zero_division=0),
        "recall": recall_score(y_test, predicted_class, zero_division=0),
        "f1": f1_score(y_test, predicted_class, zero_division=0),
        "roc_auc": roc_auc_score(y_test, predicted_probability),
        "average_precision": average_precision_score(y_test, predicted_probability),
    })

    for position, source_index in enumerate(test_indices):
        prediction_rows.append({
            "record_id": model_data.iloc[source_index]["record_id"],
            "model": model_name,
            "actual": int(y_test.iloc[position]),
            "predicted": int(predicted_class[position]),
            "predicted_probability": float(predicted_probability[position]),
        })

holdout_performance = pd.DataFrame(performance_rows).sort_values("roc_auc", ascending=False)
test_predictions = pd.DataFrame(prediction_rows)

holdout_performance.to_csv(TABLES_DIR / "holdout-performance.csv", index=False)
test_predictions.to_csv(TABLES_DIR / "test-predictions.csv", index=False)
holdout_performance.round(3)
model accuracy precision recall f1 roc_auc average_precision
1 Logistic regression 0.570 0.250 0.474 0.327 0.537 0.290
2 Decision tree 0.605 0.241 0.368 0.292 0.517 0.239
0 Dummy baseline 0.779 0.000 0.000 0.000 0.500 0.221

The holdout result estimates performance only for records similar to this fictional cohort and data-generating process. It does not establish external validity.

Inspect Confusion Matrices

Show code
fig, axes = plt.subplots(1, len(fitted_models), figsize=(14, 4))

for ax, (model_name, fitted) in zip(axes, fitted_models.items()):
    predictions = fitted.predict(X_test)
    ConfusionMatrixDisplay.from_predictions(
        y_test, predictions, display_labels=["No", "Yes"], cmap="Blues", colorbar=False, ax=ax
    )
    ax.set_title(model_name)

fig.tight_layout()
fig.savefig(FIGURES_DIR / "confusion-matrices.png", dpi=200, bbox_inches="tight")
plt.show()

Holdout confusion matrices at the default 0.50 probability threshold.

Read the cells as decisions. False negatives are actual follow-up records predicted “No.” False positives are records predicted “Yes” that were actually “No.” Their importance depends on what happens after a prediction.

Compare ROC Curves

Show code
fig, ax = plt.subplots(figsize=(7, 5.5))

for model_name, fitted in fitted_models.items():
    probabilities = fitted.predict_proba(X_test)[:, 1]
    false_positive_rate, true_positive_rate, _ = roc_curve(y_test, probabilities)
    auc_value = roc_auc_score(y_test, probabilities)
    ax.plot(false_positive_rate, true_positive_rate, label=f"{model_name} (AUC={auc_value:.2f})")

ax.plot([0, 1], [0, 1], linestyle="--", color="grey", label="Chance ranking")
ax.set(xlabel="False-positive rate", ylabel="True-positive rate", title="ROC curves on the holdout set")
ax.legend(loc="lower right")
fig.tight_layout()
fig.savefig(FIGURES_DIR / "roc-curves.png", dpi=200, bbox_inches="tight")
plt.show()

Holdout ROC curves compare ranking discrimination across thresholds.

Compare Precision–Recall Curves

Show code
fig, ax = plt.subplots(figsize=(7, 5.5))

for model_name, fitted in fitted_models.items():
    probabilities = fitted.predict_proba(X_test)[:, 1]
    precision_values, recall_values, _ = precision_recall_curve(y_test, probabilities)
    ap_value = average_precision_score(y_test, probabilities)
    ax.plot(recall_values, precision_values, label=f"{model_name} (AP={ap_value:.2f})")

ax.axhline(y_test.mean(), linestyle="--", color="grey", label=f"Prevalence ({y_test.mean():.2f})")
ax.set(xlabel="Recall", ylabel="Precision", title="Precision–recall curves on the holdout set")
ax.legend(loc="best")
fig.tight_layout()
fig.savefig(FIGURES_DIR / "precision-recall-curves.png", dpi=200, bbox_inches="tight")
plt.show()

Holdout precision–recall curves; the horizontal reference is positive prevalence.

Select a Candidate Model

Model choice should combine validation stability, relevant metrics, interpretability, complexity, and intended use. For this learning workflow, select logistic regression unless the decision tree shows a clear and stable improvement. This preference is a documented simplicity rule, not a claim that logistic regression is universally best.

Show code
selected_model_name = "Logistic regression"
selected_model = fitted_models[selected_model_name]
selected_probabilities = selected_model.predict_proba(X_test)[:, 1]

print(f"Selected candidate: {selected_model_name}")
Selected candidate: Logistic regression

Explore Decision Thresholds

The default threshold of 0.50 is conventional, not automatically optimal. Lower thresholds usually identify more positives while producing more false positives.

Show code
threshold_rows = []
for threshold in np.arange(0.10, 0.91, 0.05):
    predictions = (selected_probabilities >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, predictions, labels=[0, 1]).ravel()
    threshold_rows.append({
        "threshold": threshold,
        "precision": precision_score(y_test, predictions, zero_division=0),
        "recall": recall_score(y_test, predictions, zero_division=0),
        "f1": f1_score(y_test, predictions, zero_division=0),
        "false_positives": fp,
        "false_negatives": fn,
    })

threshold_results = pd.DataFrame(threshold_rows)
threshold_results.to_csv(TABLES_DIR / "threshold-results.csv", index=False)
threshold_results.round(3)
threshold precision recall f1 false_positives false_negatives
0 0.10 0.224 1.000 0.365 66 0
1 0.15 0.222 0.947 0.360 63 1
2 0.20 0.218 0.895 0.351 61 2
3 0.25 0.205 0.789 0.326 58 4
4 0.30 0.226 0.737 0.346 48 5
5 0.35 0.224 0.684 0.338 45 6
6 0.40 0.235 0.632 0.343 39 7
7 0.45 0.263 0.526 0.351 28 9
8 0.50 0.250 0.474 0.327 27 10
9 0.55 0.267 0.421 0.327 22 11
10 0.60 0.227 0.263 0.244 17 14
11 0.65 0.278 0.263 0.270 13 14
12 0.70 0.300 0.158 0.207 7 16
13 0.75 0.400 0.105 0.167 3 17
14 0.80 0.667 0.105 0.182 1 17
15 0.85 0.000 0.000 0.000 1 19
16 0.90 0.000 0.000 0.000 0 19
Show code
fig, ax = plt.subplots(figsize=(7.5, 5))
for metric, color in [("precision", "#036281"), ("recall", "#d97706"), ("f1", "#4f46e5")]:
    ax.plot(threshold_results["threshold"], threshold_results[metric], marker="o", label=metric.title(), color=color)
ax.axvline(0.50, linestyle="--", color="grey", label="Default threshold")
ax.set(xlabel="Probability threshold", ylabel="Metric value", ylim=(0, 1.05), title="Threshold trade-offs")
ax.legend()
fig.tight_layout()
fig.savefig(FIGURES_DIR / "threshold-tradeoff.png", dpi=200, bbox_inches="tight")
plt.show()

Precision, recall, and F1 across candidate thresholds for logistic regression.

Do not choose a threshold on the test set and then report that same result as an unbiased final estimate. In a real project, tune the threshold inside training validation and reserve a separate test set for final evaluation.

Inspect Logistic Regression Coefficients

Coefficients describe how encoded features influence the model’s log-odds, holding its other included features constant. They are not causal effects and can be unstable in small or correlated data.

Show code
feature_names = selected_model.named_steps["preprocessor"].get_feature_names_out()
coefficients = selected_model.named_steps["classifier"].coef_[0]

coefficient_table = (
    pd.DataFrame({
        "feature": feature_names,
        "coefficient": coefficients,
        "odds_multiplier": np.exp(coefficients),
    })
    .assign(absolute_coefficient=lambda frame: frame["coefficient"].abs())
    .sort_values("absolute_coefficient", ascending=False)
)

coefficient_table.to_csv(TABLES_DIR / "logistic-coefficients.csv", index=False)
coefficient_table.head(12).round(3)
feature coefficient odds_multiplier absolute_coefficient
24 categorical__under_18_Unknown -1.120 0.326 1.120
13 categorical__visit_month_11 -0.990 0.372 0.990
15 categorical__visit_month_2 -0.960 0.383 0.960
25 categorical__under_18_Yes 0.818 2.265 0.818
20 categorical__visit_month_7 0.717 2.048 0.717
3 categorical__region_East 0.675 1.963 0.675
5 categorical__region_South -0.629 0.533 0.629
8 categorical__service_type_Assessment -0.478 0.620 0.478
10 categorical__service_type_Training 0.424 1.528 0.424
11 categorical__visit_month_1 0.419 1.520 0.419
0 numeric__age 0.349 1.417 0.349
23 categorical__under_18_No 0.297 1.346 0.297

Positive coefficients move model predictions toward “Yes”; negative coefficients move them toward “No.” One-hot coefficients are relative to the model’s encoded representation and regularization. Treat them as a model diagnostic, not a ranked list of real-world causes.

Check Subgroup Performance

Aggregate performance can conceal poor performance for particular groups. This basic check compares age groups in the holdout set. It is exploratory because group counts are small.

Show code
test_context = model_data.iloc[test_indices][["under_18"]].reset_index(drop=True)
test_context["actual"] = y_test.reset_index(drop=True)
test_context["probability"] = selected_probabilities
test_context["predicted"] = (selected_probabilities >= 0.50).astype(int)

subgroup_rows = []
for group_name, group in test_context.groupby("under_18", dropna=False):
    row = {
        "group": group_name,
        "n": len(group),
        "positives": int(group["actual"].sum()),
        "accuracy": accuracy_score(group["actual"], group["predicted"]),
        "precision": precision_score(group["actual"], group["predicted"], zero_division=0),
        "recall": recall_score(group["actual"], group["predicted"], zero_division=0),
    }
    row["roc_auc"] = (
        roc_auc_score(group["actual"], group["probability"])
        if group["actual"].nunique() == 2 else np.nan
    )
    subgroup_rows.append(row)

subgroup_performance = pd.DataFrame(subgroup_rows)
subgroup_performance.to_csv(TABLES_DIR / "subgroup-performance.csv", index=False)
subgroup_performance.round(3)
group n positives accuracy precision recall roc_auc
0 No 74 15 0.622 0.276 0.533 0.617
1 Unknown 2 2 0.000 0.000 0.000 NaN
2 Yes 10 2 0.300 0.143 0.500 0.188

Do not conclude that two groups perform equivalently from similar point estimates alone. Report group sizes, outcome counts, uncertainty, relevant harms, and whether the grouping is ethically and substantively appropriate. Fairness cannot be reduced to one table.

Perform Sensitivity Checks

Repeat the workflow under defensible alternatives:

  1. remove satisfaction_score to simulate prediction before visit completion;
  2. use a temporal split, training on earlier dates and testing on later dates;
  3. compare unweighted and class-weighted logistic regression;
  4. change the random seed and inspect stability;
  5. remove the derived under_18 feature and compare results;
  6. retain reviewed records only after resolving their data-quality issues;
  7. calculate bootstrap confidence intervals when the sample supports them.

If conclusions change materially, the correct result is that performance is unstable—not that the most favorable specification should be selected.

Save the Complete Pipeline

Saving the complete fitted pipeline preserves imputation, scaling, encoding, and classification together. New data must use the same feature names and definitions.

Show code
model_path = MODELS_DIR / "logistic-follow-up-pipeline.joblib"
joblib.dump(selected_model, model_path)

reloaded_model = joblib.load(model_path)
reloaded_probabilities = reloaded_model.predict_proba(X_test)[:, 1]
np.testing.assert_allclose(selected_probabilities, reloaded_probabilities)

print(f"Saved and verified: {model_path}")
Saved and verified: results/models/logistic-follow-up-pipeline.joblib

Never load an untrusted serialized model. Files such as joblib and pickle artifacts can execute harmful code during loading.

Record the Model Decision

Show code
decision_log = pd.DataFrame([
    ["Task", "Binary classification of recorded follow-up need", "Supports a focused introductory workflow"],
    ["Split", "Stratified 75/25 random holdout", "Preserves classes; temporal and grouped alternatives remain necessary"],
    ["Baseline", "Dummy classifier using training prevalence", "Tests whether features improve on a no-signal model"],
    ["Candidate", selected_model_name, "Selected for simplicity and interpretability"],
    ["Threshold", "0.50 for reported holdout classes", "Alternative thresholds shown but not optimized on the test set"],
    ["Deployment", "Not permitted", "Small fictional data and no prospective or external validation"],
])
decision_log.columns = ["decision_area", "decision", "rationale"]
decision_log.to_csv(TABLES_DIR / "model-decision-log.csv", index=False)
decision_log
decision_area decision rationale
0 Task Binary classification of recorded follow-up need Supports a focused introductory workflow
1 Split Stratified 75/25 random holdout Preserves classes; temporal and grouped altern...
2 Baseline Dummy classifier using training prevalence Tests whether features improve on a no-signal ...
3 Candidate Logistic regression Selected for simplicity and interpretability
4 Threshold 0.50 for reported holdout classes Alternative thresholds shown but not optimized...
5 Deployment Not permitted Small fictional data and no prospective or ext...

Create a Model Card

Show code
selected_metrics = holdout_performance.loc[
    holdout_performance["model"].eq(selected_model_name)
].iloc[0]

model_card = f"""# Model Card: Follow-up Classification Learning Model



## Purpose
Educational demonstration of an end-to-end binary-classification workflow.



## Data
{len(model_data)} fictional service records after excluding unresolved review rows and invalid targets/dates.



## Target
Recorded follow-up requirement, with Yes encoded as the positive class.



## Features
Age, satisfaction score, region, service type, visit month, and derived under-18 category.



## Selected model
{selected_model_name} with mixed-type preprocessing inside the pipeline.



## Holdout results at threshold 0.50
- Accuracy: {selected_metrics['accuracy']:.3f}
- Precision: {selected_metrics['precision']:.3f}
- Recall: {selected_metrics['recall']:.3f}
- F1: {selected_metrics['f1']:.3f}
- ROC AUC: {selected_metrics['roc_auc']:.3f}
- Average precision: {selected_metrics['average_precision']:.3f}



## Limitations
Small fictional convenience dataset; uncertain target definition; random internal split only; no external, temporal, or prospective validation; unstable subgroup estimates; no impact assessment.



## Permitted use
Learning, code review, and methodological discussion.



## Prohibited use
Allocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions.
"""

(TABLES_DIR / "model-card.md").write_text(model_card, encoding="utf-8")
print(model_card)
# Model Card: Follow-up Classification Learning Model



## Purpose
Educational demonstration of an end-to-end binary-classification workflow.



## Data
343 fictional service records after excluding unresolved review rows and invalid targets/dates.



## Target
Recorded follow-up requirement, with Yes encoded as the positive class.



## Features
Age, satisfaction score, region, service type, visit month, and derived under-18 category.



## Selected model
Logistic regression with mixed-type preprocessing inside the pipeline.



## Holdout results at threshold 0.50
- Accuracy: 0.570
- Precision: 0.250
- Recall: 0.474
- F1: 0.327
- ROC AUC: 0.537
- Average precision: 0.290



## Limitations
Small fictional convenience dataset; uncertain target definition; random internal split only; no external, temporal, or prospective validation; unstable subgroup estimates; no impact assessment.



## Permitted use
Learning, code review, and methodological discussion.



## Prohibited use
Allocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions.

Interpretation and Limitations

Write a result statement that includes:

  • the prediction task and positive class;
  • cohort and holdout sizes;
  • positive prevalence;
  • baseline and candidate performance;
  • the threshold used for class predictions;
  • false-positive and false-negative counts;
  • cross-validation variability;
  • subgroup sample sizes;
  • a statement that prediction is not causation;
  • the boundary between learning and operational use.

A defensible conclusion may be that the dataset is too small or the model does not improve reliably on the baseline. That is a useful analytical finding.

Key limitations include:

  • the dataset is small, fictional, and not population-representative;
  • the target may reflect human documentation decisions rather than objective need;
  • predictors may encode access, service, or documentation patterns;
  • a random internal holdout does not test future or external performance;
  • metric estimates and subgroup comparisons have substantial uncertainty;
  • probability calibration has not been established;
  • no prospective workflow, impact assessment, or monitoring plan exists.

Export and Verify Results

Show code
expected_outputs = [
    TABLES_DIR / "feature-review.csv",
    TABLES_DIR / "cross-validation-results.csv",
    TABLES_DIR / "holdout-performance.csv",
    TABLES_DIR / "test-predictions.csv",
    TABLES_DIR / "threshold-results.csv",
    TABLES_DIR / "subgroup-performance.csv",
    TABLES_DIR / "logistic-coefficients.csv",
    TABLES_DIR / "model-decision-log.csv",
    TABLES_DIR / "model-card.md",
    FIGURES_DIR / "target-balance.png",
    FIGURES_DIR / "confusion-matrices.png",
    FIGURES_DIR / "roc-curves.png",
    FIGURES_DIR / "precision-recall-curves.png",
    FIGURES_DIR / "threshold-tradeoff.png",
    MODELS_DIR / "logistic-follow-up-pipeline.joblib",
]

output_check = pd.DataFrame({
    "path": [str(path) for path in expected_outputs],
    "exists": [path.exists() for path in expected_outputs],
})
assert output_check["exists"].all(), "One or more expected outputs were not created."
output_check
path exists
0 results/tables/feature-review.csv True
1 results/tables/cross-validation-results.csv True
2 results/tables/holdout-performance.csv True
3 results/tables/test-predictions.csv True
4 results/tables/threshold-results.csv True
5 results/tables/subgroup-performance.csv True
6 results/tables/logistic-coefficients.csv True
7 results/tables/model-decision-log.csv True
8 results/tables/model-card.md True
9 results/figures/target-balance.png True
10 results/figures/confusion-matrices.png True
11 results/figures/roc-curves.png True
12 results/figures/precision-recall-curves.png True
13 results/figures/threshold-tradeoff.png True
14 results/models/logistic-follow-up-pipeline.joblib True

Render from the VS Code terminal:

quarto render

Open products/index.html in a browser. Check navigation, layout, figures, tables, code folding, warnings, and whether all claims match the displayed results.

Portfolio Evidence

Your portfolio evidence should include:

  • the task specification and prediction moment;
  • the target and feature definitions;
  • the feature/leakage review;
  • training, cross-validation, and test design;
  • baseline and candidate comparison;
  • confusion matrix and threshold discussion;
  • reproducible pipeline code;
  • subgroup and sensitivity checks;
  • model card and decision log;
  • limitations, responsible-use boundary, and next steps;
  • a link to the repository or rendered HTML when appropriate.

Do not present only the best metric. A strong portfolio explains why the validation design is credible, what errors mean, and what evidence would be needed before use.

Project Question

Use this portfolio framing:

Can information available at the end of a service visit predict recorded follow-up need better than a prevalence-only baseline, and how stable, interpretable, and responsible is that performance?

Revise the question when working with another dataset. State the unit, outcome, prediction moment, comparison, and intended use.

Dataset

Describe what one row represents, the collection period, cohort rules, target definition, predictor availability, missingness, class balance, privacy considerations, and limits of coverage.

Do not publish direct identifiers or individual predictions. De-identification requires more than removing names; combinations of rare attributes can still identify people.

Workflow

Summarize the project as a reproducible sequence:

  1. defined the prediction unit, target, moment, user, and error costs;
  2. created a modelling cohort from quality-reviewed records;
  3. excluded identifiers and leakage-prone variables;
  4. created a stratified training and holdout split;
  5. placed preprocessing inside model pipelines;
  6. established a dummy baseline;
  7. compared logistic regression and a shallow decision tree with cross-validation;
  8. evaluated candidate models once on the holdout set;
  9. reviewed confusion matrices, probability curves, and threshold trade-offs;
  10. examined coefficients and subgroup performance cautiously;
  11. saved the complete pipeline, model card, and decision log;
  12. rendered and reviewed the final HTML.

Main Result

Report whether the selected candidate improved consistently and meaningfully on the baseline. Include the most decision-relevant metrics, cross-validation variation, error counts, and a concise limitation. If evidence is weak, say so directly.

Quality Checks

Document that:

  • row and target definitions are explicit;
  • direct identifiers and the target are absent from features;
  • the holdout set was isolated before preprocessing;
  • stratification preserved both classes;
  • transformations were fitted inside cross-validation;
  • baseline and candidate models used the same splits;
  • probability metrics used probabilities rather than class labels;
  • confusion-matrix labels and positive class are correct;
  • test-set threshold tuning was not reported as unbiased performance;
  • subgroup tables include sample and positive counts;
  • the saved pipeline reproduces the original probabilities;
  • exported outputs exist and the HTML was inspected.

Limitations

Explain dataset, target, validation, modelling, fairness, and deployment limitations separately. State what additional data and evaluation would change your confidence.

Tools

List only tools used in the completed project. The standard workflow uses Python, pandas, numpy, matplotlib, scikit-learn, joblib, VS Code, Quarto, and optionally Git and GitHub for version control and sharing.

Reflection and Next Steps

Write brief responses to these prompts:

  1. What exactly is known at the prediction moment?
  2. Which excluded variable posed the clearest leakage or privacy risk?
  3. Did either candidate improve consistently on the dummy baseline?
  4. Which metric best reflected the practical cost of errors?
  5. How did threshold changes affect false negatives and false positives?
  6. How stable were results across cross-validation folds?
  7. What did the subgroup check reveal, and what could it not establish?
  8. Which coefficient was easiest to misinterpret causally?
  9. Would a temporal or grouped split be more realistic for your own data?
  10. What evidence would be required before any operational pilot?

The next practical can extend this foundation into model tuning, calibration, explainability, temporal validation, or monitoring—but additional complexity should answer a defined need.

Completion Checklist

Before marking the practical complete, confirm that you have:

Practical Complete

You have completed Data Science Practical 008: Introduction to Machine Learning Practical.

You now have a reproducible introductory classification workflow, an explicit validation design, a baseline and candidate comparison, diagnostic figures and tables, a saved end-to-end pipeline, and portfolio evidence demonstrating responsible machine-learning practice.

Keep the input data, complete practical.qmd, exported evidence, model card, decision log, and rendered HTML together. They provide a traceable record from prediction question to model evaluation—and a reminder that a fitted model is only one part of a trustworthy decision process.

© Complex Data Insights | CDI Practicals

Hands-on exercises and step-by-step tools