A hands-on exercise for building, evaluating, and communicating a responsible classification model with Python
Author
Complex Data Insights
Published
Aug 2026
COMPLEX DATA INSIGHTS
CDI Practicals
Hands-on exercises and step-by-step tools for doing, applying, and building evidence.
Introduction to Machine Learning Practical
A hands-on exercise for building, evaluating, and communicating a responsible classification model with Python
AuthorComplex Data Insights
PublishedAug 2026
ID CDI-DATASCIENCE-PRACTICAL-008
Type Practical Product
Audience Students, researchers, analysts, professionals, mentors, and learners developing practical data science skills
ThemeUseful machine learning begins with a clear prediction task, a defensible validation design, an honest baseline, and evidence-aware communication
How to Use This Practical
Complete this practical actively. Run every code block, inspect every result, and write your own interpretation before moving on. The example uses a small fictional service-delivery dataset so you can focus on the workflow rather than computing power.
You are encouraged to:
define the prediction moment and target before choosing a model;
separate training from testing before learning transformations;
compare every model with a simple baseline;
evaluate more than accuracy;
treat predicted probabilities as estimates, not facts;
check whether performance differs across meaningful groups;
document uncertainty, limitations, and intended use;
avoid deploying a model whose consequences have not been assessed.
Browser-first workflow
Follow the rendered HTML practical in a browser while working in VS Code. Use VS Code to run, adapt, and save the Python workflow. JupyterLab, Positron, or Google Colab can also be used, but repository paths and the final Quarto render are designed for VS Code.
Recommended Approach
Complete the practical in four passes.
Pass 1: Frame the task
Define the outcome, unit of prediction, prediction moment, intended user, and cost of errors. Decide whether machine learning is appropriate.
Pass 2: Build a valid comparison
Prepare features without leakage, create a stratified holdout set, establish a dummy baseline, and train two understandable classifiers.
Pass 3: Evaluate and stress-test
Use cross-validation and holdout metrics, inspect the confusion matrix and probabilities, review thresholds, and compare subgroup performance.
Pass 4: Communicate and preserve
Record model decisions, limitations, and non-permitted uses. Export results, save the complete pipeline, render the practical, and inspect the HTML.
Practical Overview
Machine learning uses patterns in existing data to make predictions for new observations. This practical introduces supervised binary classification: each historical record has features and a known two-category outcome, and the model learns to estimate the probability of one category.
This practical begins the Machine Learning and Systems sequence and builds on the earlier Data Foundations and Databases and Pipelines practicals. You will use cleaned service-delivery records to explore a narrow operational question:
Can information available at the end of a service visit help identify records that may require follow-up?
You will compare a prevalence-only baseline, logistic regression, and a shallow decision tree. The goal is not to produce a deployable service-allocation system. It is to learn a reproducible modelling workflow and recognize when the evidence is too limited for operational use.
Learning Outcomes
By completing this practical, you will be able to:
distinguish prediction from explanation and causal inference;
define a target, features, observation unit, and prediction moment;
identify identifier, proxy, temporal, and preprocessing leakage;
create reproducible stratified training and test sets;
build mixed-type preprocessing inside a scikit-learn pipeline;
establish a dummy baseline;
train and compare logistic regression and decision-tree classifiers;
use cross-validation without touching the test set;
interpret accuracy, precision, recall, F1, ROC AUC, and confusion matrices;
examine predicted probabilities and decision thresholds;
perform a basic subgroup performance check;
save a fitted pipeline and portfolio-ready evidence;
communicate limitations and responsible-use boundaries.
What You Will Produce
By the end of the practical, you will create:
a machine-learning problem statement;
a feature and leakage review;
reproducible training and test sets;
a baseline and two candidate models;
cross-validation results;
a holdout performance table;
confusion-matrix, ROC, precision–recall, and threshold figures;
Open the repository root in VS Code so relative paths work consistently.
Dataset and Prediction Context
One row represents one fictional service record. Candidate predictors include region, service type, participant age, visit month, satisfaction score, and whether the participant was under 18. The binary target is follow_up_required.
The target balance and sample size will strongly affect evaluation. A model can appear accurate simply by predicting the common class. The records are fictional and do not represent a population, intervention, or validated operational process.
Direct identifiers—record ID, name, and email—are excluded. A direct identifier may let a flexible model memorize records without learning a generalizable relationship. Names and emails also create unnecessary privacy risk.
Learning exercise, not a deployment model
Do not use this example model to allocate care, deny services, rank people, or automate follow-up. Deployment would require representative data, an independently validated outcome, domain review, privacy and impact assessment, prospective validation, monitoring, and a clear human decision process.
Define the Prediction Task
Complete this specification before coding:
Component
Example definition
Unit
One completed service visit
Target
Whether follow-up is recorded as required (Yes or No)
Positive class
Yes
Prediction moment
At the end of the visit, after satisfaction is recorded
Intended user
Analyst evaluating whether a predictive workflow is feasible
Intended action
No operational action; methodological learning only
More costly error
A false negative may miss a record that needs review
Validation aim
Estimate performance on unseen records from the same fictional process
The prediction moment matters. If prediction were required before the visit ended, satisfaction score would not yet be available and must be removed.
Prediction Is Not Explanation
A predictive association does not establish why follow-up is required. A model may exploit documentation habits, service availability, missingness, or historical decisions. Even strong test performance would not show that changing a predictor would change the outcome.
Use predictive language:
“the model assigned a higher estimated probability”;
“this feature was associated with predictions in this dataset”;
“holdout performance suggests limited or useful discrimination.”
Avoid causal language such as “age causes follow-up” or “the model proves which service creates need.”
Records retained for review may contain unresolved dates, categories, or values. Excluding them defines the modelling population more clearly; it does not make the remaining sample representative. The source also contains reused record_id values attached to different people or visits. Because an ambiguous identifier can compromise row-level traceability and grouped validation, those duplicate-ID rows are excluded from this learning cohort and counted in the cohort summary.
Define the Modelling Cohort
Show code
model_data = data.loc[data["requires_review"].eq(False)].copy()model_data = model_data.loc[model_data["follow_up_required"].isin(["Yes", "No"])].copy()model_data["visit_date"] = pd.to_datetime(model_data["visit_date"], errors="coerce")model_data = model_data.loc[model_data["visit_date"].notna()].copy()model_data["target"] = model_data["follow_up_required"].map({"No": 0, "Yes": 1}).astype(int)model_data["visit_month"] = model_data["visit_date"].dt.month.astype(str)model_data["under_18"] = np.where( model_data["age"].isna(), "Unknown", np.where(model_data["age"] <18, "Yes", "No"))# A record identifier should identify one observation. The source file contains# two identifiers that are reused for different people/visits, so retain neither# ambiguous row in the modelling cohort and record the exclusion explicitly.duplicate_id_mask = model_data["record_id"].duplicated(keep=False)duplicate_id_rows =int(duplicate_id_mask.sum())duplicate_id_values =sorted(model_data.loc[duplicate_id_mask, "record_id"].unique())model_data = model_data.loc[~duplicate_id_mask].copy()assert model_data["record_id"].is_unique, "Record IDs must be unique after exclusions."assert model_data["target"].nunique() ==2, "Both target classes are required."cohort_summary = pd.Series({"source_rows": len(data),"duplicate_id_rows_excluded": duplicate_id_rows,"modelling_rows": len(model_data),"positive_rows": int(model_data["target"].sum()),"positive_prevalence": model_data["target"].mean(),})cohort_summary
target_counts = model_data["follow_up_required"].value_counts().reindex(["No", "Yes"])fig, ax = plt.subplots(figsize=(6.5, 4.2))ax.bar(target_counts.index, target_counts.values, color="#036281")ax.set(title="Follow-up requirement in the modelling cohort", xlabel="Follow-up required", ylabel="Records")for index, value inenumerate(target_counts.values): ax.text(index, value +0.5, str(value), ha="center")fig.tight_layout()fig.savefig(FIGURES_DIR /"target-balance.png", dpi=200, bbox_inches="tight")plt.show()
Follow-up target balance in the modelling cohort.
Class imbalance changes what “good” performance means. Record the positive prevalence; it supplies context for the dummy baseline and precision–recall curve.
Review Features and Leakage
Show code
feature_review = pd.DataFrame([ ["age", "numeric", "Include", "Available at prediction time; may be missing"], ["satisfaction_score", "numeric", "Include conditionally", "Only valid if prediction occurs after scoring"], ["region", "categorical", "Include", "May encode structural or access differences; check subgroups"], ["service_type", "categorical", "Include", "Available at the end of the visit"], ["visit_month", "categorical", "Include", "Derived from visit date; captures seasonality, not identity"], ["under_18", "categorical", "Include", "Derived from age for an explicit age-group check"], ["record_id", "identifier", "Exclude", "Identifier; duplicate IDs are excluded from the modelling cohort and identifiers are never used as features"], ["participant_name", "identifier", "Exclude", "Privacy and memorization risk"], ["contact_email", "identifier", "Exclude", "Privacy and memorization risk"], ["follow_up_required", "target", "Exclude from features", "Direct target leakage"], ["requires_review", "post-cleaning flag", "Exclude", "Used to define cohort; may encode data-quality process"],])feature_review.columns = ["variable", "role", "decision", "reason"]feature_review.to_csv(TABLES_DIR /"feature-review.csv", index=False)feature_review
variable
role
decision
reason
0
age
numeric
Include
Available at prediction time; may be missing
1
satisfaction_score
numeric
Include conditionally
Only valid if prediction occurs after scoring
2
region
categorical
Include
May encode structural or access differences; c...
3
service_type
categorical
Include
Available at the end of the visit
4
visit_month
categorical
Include
Derived from visit date; captures seasonality,...
5
under_18
categorical
Include
Derived from age for an explicit age-group check
6
record_id
identifier
Exclude
Identifier; duplicate IDs are excluded from th...
7
participant_name
identifier
Exclude
Privacy and memorization risk
8
contact_email
identifier
Exclude
Privacy and memorization risk
9
follow_up_required
target
Exclude from features
Direct target leakage
10
requires_review
post-cleaning flag
Exclude
Used to define cohort; may encode data-quality...
Leakage occurs when training uses information that would not be legitimately available for a new prediction. It can enter through target-derived columns, future information, preprocessing performed before splitting, repeated people across splits, or an unrealistic random split for time-dependent data.
Use group-aware splitting when one person contributes multiple records. Use a temporal split when the real task predicts future periods from earlier periods. A random split can overstate performance when related or future observations appear in both sets.
Build Preprocessing Pipelines
Numeric values are median-imputed and standardized. Categorical values are most-frequent-imputed and one-hot encoded. All transformations live inside the pipeline, so they are learned from training folds only.
Do not impute, scale, or select categories using the complete dataset before splitting. Those apparently small actions allow test information to influence training.
Establish a Baseline
The dummy model predicts according to the training prevalence and uses no features. A candidate model that cannot improve meaningfully on this baseline has not demonstrated useful signal.
The holdout result estimates performance only for records similar to this fictional cohort and data-generating process. It does not establish external validity.
Holdout confusion matrices at the default 0.50 probability threshold.
Read the cells as decisions. False negatives are actual follow-up records predicted “No.” False positives are records predicted “Yes” that were actually “No.” Their importance depends on what happens after a prediction.
Holdout precision–recall curves; the horizontal reference is positive prevalence.
Select a Candidate Model
Model choice should combine validation stability, relevant metrics, interpretability, complexity, and intended use. For this learning workflow, select logistic regression unless the decision tree shows a clear and stable improvement. This preference is a documented simplicity rule, not a claim that logistic regression is universally best.
The default threshold of 0.50 is conventional, not automatically optimal. Lower thresholds usually identify more positives while producing more false positives.
Precision, recall, and F1 across candidate thresholds for logistic regression.
Do not choose a threshold on the test set and then report that same result as an unbiased final estimate. In a real project, tune the threshold inside training validation and reserve a separate test set for final evaluation.
Inspect Logistic Regression Coefficients
Coefficients describe how encoded features influence the model’s log-odds, holding its other included features constant. They are not causal effects and can be unstable in small or correlated data.
Positive coefficients move model predictions toward “Yes”; negative coefficients move them toward “No.” One-hot coefficients are relative to the model’s encoded representation and regularization. Treat them as a model diagnostic, not a ranked list of real-world causes.
Check Subgroup Performance
Aggregate performance can conceal poor performance for particular groups. This basic check compares age groups in the holdout set. It is exploratory because group counts are small.
Do not conclude that two groups perform equivalently from similar point estimates alone. Report group sizes, outcome counts, uncertainty, relevant harms, and whether the grouping is ethically and substantively appropriate. Fairness cannot be reduced to one table.
Perform Sensitivity Checks
Repeat the workflow under defensible alternatives:
remove satisfaction_score to simulate prediction before visit completion;
use a temporal split, training on earlier dates and testing on later dates;
compare unweighted and class-weighted logistic regression;
change the random seed and inspect stability;
remove the derived under_18 feature and compare results;
retain reviewed records only after resolving their data-quality issues;
calculate bootstrap confidence intervals when the sample supports them.
If conclusions change materially, the correct result is that performance is unstable—not that the most favorable specification should be selected.
Save the Complete Pipeline
Saving the complete fitted pipeline preserves imputation, scaling, encoding, and classification together. New data must use the same feature names and definitions.
Saved and verified: results/models/logistic-follow-up-pipeline.joblib
Never load an untrusted serialized model. Files such as joblib and pickle artifacts can execute harmful code during loading.
Record the Model Decision
Show code
decision_log = pd.DataFrame([ ["Task", "Binary classification of recorded follow-up need", "Supports a focused introductory workflow"], ["Split", "Stratified 75/25 random holdout", "Preserves classes; temporal and grouped alternatives remain necessary"], ["Baseline", "Dummy classifier using training prevalence", "Tests whether features improve on a no-signal model"], ["Candidate", selected_model_name, "Selected for simplicity and interpretability"], ["Threshold", "0.50 for reported holdout classes", "Alternative thresholds shown but not optimized on the test set"], ["Deployment", "Not permitted", "Small fictional data and no prospective or external validation"],])decision_log.columns = ["decision_area", "decision", "rationale"]decision_log.to_csv(TABLES_DIR /"model-decision-log.csv", index=False)decision_log
decision_area
decision
rationale
0
Task
Binary classification of recorded follow-up need
Supports a focused introductory workflow
1
Split
Stratified 75/25 random holdout
Preserves classes; temporal and grouped altern...
2
Baseline
Dummy classifier using training prevalence
Tests whether features improve on a no-signal ...
3
Candidate
Logistic regression
Selected for simplicity and interpretability
4
Threshold
0.50 for reported holdout classes
Alternative thresholds shown but not optimized...
5
Deployment
Not permitted
Small fictional data and no prospective or ext...
Create a Model Card
Show code
selected_metrics = holdout_performance.loc[ holdout_performance["model"].eq(selected_model_name)].iloc[0]model_card =f"""# Model Card: Follow-up Classification Learning Model## PurposeEducational demonstration of an end-to-end binary-classification workflow.## Data{len(model_data)} fictional service records after excluding unresolved review rows and invalid targets/dates.## TargetRecorded follow-up requirement, with Yes encoded as the positive class.## FeaturesAge, satisfaction score, region, service type, visit month, and derived under-18 category.## Selected model{selected_model_name} with mixed-type preprocessing inside the pipeline.## Holdout results at threshold 0.50- Accuracy: {selected_metrics['accuracy']:.3f}- Precision: {selected_metrics['precision']:.3f}- Recall: {selected_metrics['recall']:.3f}- F1: {selected_metrics['f1']:.3f}- ROC AUC: {selected_metrics['roc_auc']:.3f}- Average precision: {selected_metrics['average_precision']:.3f}## LimitationsSmall fictional convenience dataset; uncertain target definition; random internal split only; no external, temporal, or prospective validation; unstable subgroup estimates; no impact assessment.## Permitted useLearning, code review, and methodological discussion.## Prohibited useAllocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions."""(TABLES_DIR /"model-card.md").write_text(model_card, encoding="utf-8")print(model_card)
# Model Card: Follow-up Classification Learning Model
## Purpose
Educational demonstration of an end-to-end binary-classification workflow.
## Data
343 fictional service records after excluding unresolved review rows and invalid targets/dates.
## Target
Recorded follow-up requirement, with Yes encoded as the positive class.
## Features
Age, satisfaction score, region, service type, visit month, and derived under-18 category.
## Selected model
Logistic regression with mixed-type preprocessing inside the pipeline.
## Holdout results at threshold 0.50
- Accuracy: 0.570
- Precision: 0.250
- Recall: 0.474
- F1: 0.327
- ROC AUC: 0.537
- Average precision: 0.290
## Limitations
Small fictional convenience dataset; uncertain target definition; random internal split only; no external, temporal, or prospective validation; unstable subgroup estimates; no impact assessment.
## Permitted use
Learning, code review, and methodological discussion.
## Prohibited use
Allocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions.
Interpretation and Limitations
Write a result statement that includes:
the prediction task and positive class;
cohort and holdout sizes;
positive prevalence;
baseline and candidate performance;
the threshold used for class predictions;
false-positive and false-negative counts;
cross-validation variability;
subgroup sample sizes;
a statement that prediction is not causation;
the boundary between learning and operational use.
A defensible conclusion may be that the dataset is too small or the model does not improve reliably on the baseline. That is a useful analytical finding.
Key limitations include:
the dataset is small, fictional, and not population-representative;
the target may reflect human documentation decisions rather than objective need;
predictors may encode access, service, or documentation patterns;
a random internal holdout does not test future or external performance;
metric estimates and subgroup comparisons have substantial uncertainty;
probability calibration has not been established;
no prospective workflow, impact assessment, or monitoring plan exists.
Export and Verify Results
Show code
expected_outputs = [ TABLES_DIR /"feature-review.csv", TABLES_DIR /"cross-validation-results.csv", TABLES_DIR /"holdout-performance.csv", TABLES_DIR /"test-predictions.csv", TABLES_DIR /"threshold-results.csv", TABLES_DIR /"subgroup-performance.csv", TABLES_DIR /"logistic-coefficients.csv", TABLES_DIR /"model-decision-log.csv", TABLES_DIR /"model-card.md", FIGURES_DIR /"target-balance.png", FIGURES_DIR /"confusion-matrices.png", FIGURES_DIR /"roc-curves.png", FIGURES_DIR /"precision-recall-curves.png", FIGURES_DIR /"threshold-tradeoff.png", MODELS_DIR /"logistic-follow-up-pipeline.joblib",]output_check = pd.DataFrame({"path": [str(path) for path in expected_outputs],"exists": [path.exists() for path in expected_outputs],})assert output_check["exists"].all(), "One or more expected outputs were not created."output_check
path
exists
0
results/tables/feature-review.csv
True
1
results/tables/cross-validation-results.csv
True
2
results/tables/holdout-performance.csv
True
3
results/tables/test-predictions.csv
True
4
results/tables/threshold-results.csv
True
5
results/tables/subgroup-performance.csv
True
6
results/tables/logistic-coefficients.csv
True
7
results/tables/model-decision-log.csv
True
8
results/tables/model-card.md
True
9
results/figures/target-balance.png
True
10
results/figures/confusion-matrices.png
True
11
results/figures/roc-curves.png
True
12
results/figures/precision-recall-curves.png
True
13
results/figures/threshold-tradeoff.png
True
14
results/models/logistic-follow-up-pipeline.joblib
True
Render from the VS Code terminal:
quarto render
Open products/index.html in a browser. Check navigation, layout, figures, tables, code folding, warnings, and whether all claims match the displayed results.
Portfolio Evidence
Your portfolio evidence should include:
the task specification and prediction moment;
the target and feature definitions;
the feature/leakage review;
training, cross-validation, and test design;
baseline and candidate comparison;
confusion matrix and threshold discussion;
reproducible pipeline code;
subgroup and sensitivity checks;
model card and decision log;
limitations, responsible-use boundary, and next steps;
a link to the repository or rendered HTML when appropriate.
Do not present only the best metric. A strong portfolio explains why the validation design is credible, what errors mean, and what evidence would be needed before use.
Project Question
Use this portfolio framing:
Can information available at the end of a service visit predict recorded follow-up need better than a prevalence-only baseline, and how stable, interpretable, and responsible is that performance?
Revise the question when working with another dataset. State the unit, outcome, prediction moment, comparison, and intended use.
Dataset
Describe what one row represents, the collection period, cohort rules, target definition, predictor availability, missingness, class balance, privacy considerations, and limits of coverage.
Do not publish direct identifiers or individual predictions. De-identification requires more than removing names; combinations of rare attributes can still identify people.
Workflow
Summarize the project as a reproducible sequence:
defined the prediction unit, target, moment, user, and error costs;
created a modelling cohort from quality-reviewed records;
excluded identifiers and leakage-prone variables;
created a stratified training and holdout split;
placed preprocessing inside model pipelines;
established a dummy baseline;
compared logistic regression and a shallow decision tree with cross-validation;
evaluated candidate models once on the holdout set;
reviewed confusion matrices, probability curves, and threshold trade-offs;
examined coefficients and subgroup performance cautiously;
saved the complete pipeline, model card, and decision log;
rendered and reviewed the final HTML.
Main Result
Report whether the selected candidate improved consistently and meaningfully on the baseline. Include the most decision-relevant metrics, cross-validation variation, error counts, and a concise limitation. If evidence is weak, say so directly.
Quality Checks
Document that:
row and target definitions are explicit;
direct identifiers and the target are absent from features;
the holdout set was isolated before preprocessing;
stratification preserved both classes;
transformations were fitted inside cross-validation;
baseline and candidate models used the same splits;
probability metrics used probabilities rather than class labels;
confusion-matrix labels and positive class are correct;
test-set threshold tuning was not reported as unbiased performance;
subgroup tables include sample and positive counts;
the saved pipeline reproduces the original probabilities;
exported outputs exist and the HTML was inspected.
Limitations
Explain dataset, target, validation, modelling, fairness, and deployment limitations separately. State what additional data and evaluation would change your confidence.
Tools
List only tools used in the completed project. The standard workflow uses Python, pandas, numpy, matplotlib, scikit-learn, joblib, VS Code, Quarto, and optionally Git and GitHub for version control and sharing.
Reflection and Next Steps
Write brief responses to these prompts:
What exactly is known at the prediction moment?
Which excluded variable posed the clearest leakage or privacy risk?
Did either candidate improve consistently on the dummy baseline?
Which metric best reflected the practical cost of errors?
How did threshold changes affect false negatives and false positives?
How stable were results across cross-validation folds?
What did the subgroup check reveal, and what could it not establish?
Which coefficient was easiest to misinterpret causally?
Would a temporal or grouped split be more realistic for your own data?
What evidence would be required before any operational pilot?
The next practical can extend this foundation into model tuning, calibration, explainability, temporal validation, or monitoring—but additional complexity should answer a defined need.
Completion Checklist
Before marking the practical complete, confirm that you have:
Practical Complete
You have completed Data Science Practical 008: Introduction to Machine Learning Practical.
You now have a reproducible introductory classification workflow, an explicit validation design, a baseline and candidate comparison, diagnostic figures and tables, a saved end-to-end pipeline, and portfolio evidence demonstrating responsible machine-learning practice.
Keep the input data, complete practical.qmd, exported evidence, model card, decision log, and rendered HTML together. They provide a traceable record from prediction question to model evaluation—and a reminder that a fitted model is only one part of a trustworthy decision process.