A hands-on exercise for engineering features, tuning pipelines, and selecting a robust classification model with Python
Author
Complex Data Insights
Published
Aug 2026
COMPLEX DATA INSIGHTS
CDI Practicals
Hands-on exercises and step-by-step tools for doing, applying, and building evidence.
Model Building Practical
A hands-on exercise for engineering features, tuning pipelines, and selecting a robust classification model with Python
AuthorComplex Data Insights
PublishedAug 2026
ID CDI-DATASCIENCE-PRACTICAL-005
Type Practical Product
Audience Students, researchers, analysts, professionals, mentors, and learners developing practical data science skills
Theme Strong model building improves a defensible baseline through purposeful features, leakage-safe tuning, repeated validation, and honest model selection
How to Use This Practical
Complete this practical actively. Run each code block, inspect every result, and record what changed before moving on. The aim is not to obtain the largest score. The aim is to make model-development choices that can be explained, reproduced, and challenged.
You are encouraged to:
begin from the task and baseline established in Practical 004;
create features only when their meaning and availability are defensible;
keep every learned transformation inside a pipeline;
tune models using training data only;
separate model selection from final holdout evaluation;
examine discrimination, calibration, errors, and stability together;
prefer a simpler model when added complexity has no reliable benefit;
document rejected ideas as carefully as the selected model.
Browser-first workflow
Follow the rendered HTML practical in a browser while working in VS Code. Use VS Code to run, adapt, and save the Python workflow. JupyterLab, Positron, or Google Colab can also be used, but repository paths and the final Quarto render are designed for VS Code.
Recommended Approach
Complete the practical in four passes.
Pass 1: Establish the design
Restate the prediction task, freeze the holdout set, define the primary metric, and write down the candidate feature ideas before tuning anything.
Pass 2: Build candidates
Create a base feature set and an engineered feature set. Place preprocessing and models inside pipelines, then define small, purposeful search spaces.
Pass 3: Select and diagnose
Use nested cross-validation to compare candidates without treating tuning results as unbiased performance. Examine variability, learning curves, calibration, feature importance, and errors.
Pass 4: Finalize and communicate
Evaluate the selected specification once on the untouched holdout set. Save the complete pipeline, development evidence, model report, and portfolio-ready summary.
Practical Overview
Practical 004 introduced the complete supervised-classification workflow. This practical concentrates on the next layer of judgement: improving a model without weakening the evidence supporting it.
You will revisit the fictional service-delivery question:
Can information available at the end of a service visit help identify records that may require follow-up?
You will compare a regularized logistic regression with a random forest. Both candidates will be evaluated as complete pipelines. You will add purposeful features, tune a limited set of hyperparameters, estimate selection performance with nested cross-validation, and diagnose the selected model.
The dataset is intentionally small. That makes instability visible and creates an important constraint: complex searches and confident claims are not justified.
Learning Outcomes
By completing this practical, you will be able to:
distinguish feature construction, feature transformation, and feature selection;
create domain-informed features without using the target;
prevent leakage during preprocessing and hyperparameter tuning;
You need Python 3.10 or later, VS Code with the Python and Jupyter extensions, Quarto, and these packages:
python-m pip install -r requirements.txt
The practical uses:
data/cleaned-service-data.csv
Keep the input unchanged. All derived tables, figures, and models belong under results/.
The supplied project files also include a standalone, script-first workflow. After installing the requirements, run:
bash scripts/bash/run-workflow.sh
This regenerates the deterministic synthetic dataset, executes the complete model-development workflow, verifies the fitted pipeline, and writes an audit manifest. The embedded code remains available for step-by-step learning; the standalone scripts provide a reproducible end-to-end check.
Open the repository root in VS Code so relative paths work consistently.
Relationship to Practical 004
Practical 004 established the task, cohort, baseline, initial feature set, candidate models, holdout principle, and responsible-use boundary. Practical 005 does not replace those foundations.
The progression is:
Practical 004
Practical 005
Compare simple untuned candidates
Tune limited candidate pipelines
Use one cross-validation procedure
Separate tuning from performance estimation
Inspect initial features
Engineer and document new features
Evaluate discrimination and thresholds
Add calibration, learning curves, and error analysis
Choose an introductory candidate
Make a structured model-selection decision
If you have not completed Practical 004, read its sections on prediction framing, leakage, preprocessing pipelines, baselines, metrics, and holdout evaluation before continuing.
Dataset and Modelling Context
One row represents one fictional service record. The target is follow_up_required. Candidate information available at the end of a visit includes region, service type, age, visit date, and satisfaction score.
Direct identifiers, contact fields, the review flag, and the original target label are not predictors. Engineered features must be computable at the stated prediction moment. They must not summarize future records or use the target.
Learning exercise, not an operational model
Do not use this model to allocate or deny services, rank people, or automate follow-up. Operational use would require representative data, a validated target, domain and community review, privacy and impact assessment, external and prospective validation, monitoring, and a defined human decision process.
Define the Model-Development Brief
Freeze the rules before searching for a better model.
Component
Decision for this practical
Unit
One completed service visit
Positive class
Follow-up required (Yes)
Prediction moment
End of visit, after satisfaction is recorded
Primary selection metric
ROC AUC
Supporting metrics
Average precision, recall, precision, F1, Brier score
Holdout rule
Isolate 20% before feature comparison or tuning
Candidate families
Regularized logistic regression and random forest
Complexity rule
Prefer the simpler model when performance is practically similar
The holdout set is not a workspace for feature selection, tuning, threshold choice, or repeated comparison. It represents the final internal check after the specification has been selected.
Feature engineering should express a plausible pattern more directly, improve measurement, or make a model easier to estimate. It should not be a search for accidental correlations.
For this exercise, create:
age_band: a coarse life-stage category;
visit_quarter: a broader seasonal representation than twelve months;
satisfaction_missing: whether satisfaction was unrecorded;
age_missing: whether age was unrecorded;
low_satisfaction: whether the recorded score is 1 or 2;
age_satisfaction: a numerical interaction between age and satisfaction.
The missingness indicators allow the model to distinguish an imputed value from a genuinely observed central value. Their usefulness remains an empirical question, and their meaning may reflect documentation practices rather than participant characteristics.
Show code
feature_plan = pd.DataFrame([ ["age_band", "age", "Categorical", "Represent broad non-linear age patterns", "Available at prediction time"], ["visit_quarter", "visit_date", "Categorical", "Represent broad seasonality", "May encode operational change"], ["satisfaction_missing", "satisfaction_score", "Categorical", "Preserve information about missingness", "May encode documentation practice"], ["age_missing", "age", "Categorical", "Preserve information about missingness", "May encode documentation practice"], ["low_satisfaction", "satisfaction_score", "Categorical", "Represent a potentially useful threshold", "Threshold chosen before outcome review"], ["age_satisfaction", "age and satisfaction_score", "Numerical", "Represent a simple interaction", "Difficult to interpret in isolation"],], columns=["feature", "source", "type", "rationale", "caution"])feature_plan.to_csv(TABLES_DIR /"feature-plan.csv", index=False)feature_plan
feature
source
type
rationale
caution
0
age_band
age
Categorical
Represent broad non-linear age patterns
Available at prediction time
1
visit_quarter
visit_date
Categorical
Represent broad seasonality
May encode operational change
2
satisfaction_missing
satisfaction_score
Categorical
Preserve information about missingness
May encode documentation practice
3
age_missing
age
Categorical
Preserve information about missingness
May encode documentation practice
4
low_satisfaction
satisfaction_score
Categorical
Represent a potentially useful threshold
Threshold chosen before outcome review
5
age_satisfaction
age and satisfaction_score
Numerical
Represent a simple interaction
Difficult to interpret in isolation
Construct Features Reproducibly
The function below uses only columns available at prediction time. It does not learn from the target or from population-level summaries. Learned operations—imputation, scaling, and category encoding—remain inside the candidate pipelines.
feature_dictionary = pd.DataFrame([ ["region", "Categorical", "Source", "Service region"], ["service_type", "Categorical", "Source", "Type of service visit"], ["age", "Numerical", "Source", "Participant age in years"], ["satisfaction_score", "Numerical", "Source", "End-of-visit score"], ["visit_month", "Categorical", "Derived", "Calendar month of visit"], ["visit_quarter", "Categorical", "Derived", "Calendar quarter of visit"], ["age_band", "Categorical", "Derived", "Predefined broad age band"], ["age_missing", "Categorical", "Derived", "Age missingness indicator"], ["satisfaction_missing", "Categorical", "Derived", "Satisfaction missingness indicator"], ["low_satisfaction", "Categorical", "Derived", "Score of 1 or 2; Unknown if missing"], ["age_satisfaction", "Numerical", "Derived", "Age multiplied by satisfaction score"],], columns=["feature", "type", "origin", "definition"])feature_dictionary.to_csv(TABLES_DIR /"feature-dictionary.csv", index=False)feature_dictionary
feature
type
origin
definition
0
region
Categorical
Source
Service region
1
service_type
Categorical
Source
Type of service visit
2
age
Numerical
Source
Participant age in years
3
satisfaction_score
Numerical
Source
End-of-visit score
4
visit_month
Categorical
Derived
Calendar month of visit
5
visit_quarter
Categorical
Derived
Calendar quarter of visit
6
age_band
Categorical
Derived
Predefined broad age band
7
age_missing
Categorical
Derived
Age missingness indicator
8
satisfaction_missing
Categorical
Derived
Satisfaction missingness indicator
9
low_satisfaction
Categorical
Derived
Score of 1 or 2; Unknown if missing
10
age_satisfaction
Numerical
Derived
Age multiplied by satisfaction score
Define Base and Engineered Feature Sets
The base set approximates the introductory model. The engineered set adds the planned representations. Comparing them tests whether the additional complexity improves development performance consistently.
Imputation, scaling, and one-hot encoding are learned from the training partition within each resample. handle_unknown="ignore" allows a category seen later to be represented without breaking prediction.
Logistic regression offers a regularized linear probability model. The random forest can represent interactions and non-linearities, but has more flexibility and may be unstable in a small dataset.
Large searches can overfit the resampling procedure, consume time, and create a false impression of precision. Use a small grid tied to understandable modelling choices.
Show code
search_spaces = {"logistic_regression": {"model__C": [0.01, 0.1, 1.0, 10.0],"model__class_weight": [None, "balanced"], },"random_forest": {"model__n_estimators": [200],"model__max_depth": [2, 4, None],"model__min_samples_leaf": [2, 5, 10],"model__max_features": ["sqrt", 0.75],"model__class_weight": [None, "balanced"], },}pd.Series({name: np.prod([len(v) for v in grid.values()]) for name, grid in search_spaces.items()}, name="combinations")
Ordinary grid search reports the best average score among many tried settings. That score helped choose the setting, so it is not an independent estimate of the selected procedure.
Nested cross-validation separates two jobs:
the inner loop selects hyperparameters;
the outer loop estimates how the complete selection procedure performs on unseen development folds.
The untouched holdout set remains outside both loops.
Run Nested Cross-Validation
Use identical outer splits for both model families. With a small dataset, report the individual outer-fold scores and their spread—not only their mean.
After estimating the selection procedure, fit each search to the complete development set. These searches define the candidate specifications to compare before the holdout is opened.
Use a tolerance rule to avoid choosing complexity based on a negligible estimated difference. Here, the random forest must exceed logistic regression by at least 0.02 mean outer-fold ROC AUC. Otherwise, select logistic regression.
Show code
mean_auc = nested_summary.set_index("model")["mean_roc_auc"]complexity_margin =0.02if mean_auc["random_forest"] >= mean_auc["logistic_regression"] + complexity_margin: selected_model_name ="random_forest" selection_reason ="Random forest exceeded the predeclared complexity margin."else: selected_model_name ="logistic_regression" selection_reason ="Random forest did not exceed the predeclared complexity margin; prefer logistic regression."selected_pipeline = final_searches[selected_model_name].best_estimator_print(selected_model_name)print(selection_reason)
logistic_regression
Random forest did not exceed the predeclared complexity margin; prefer logistic regression.
This rule is instructional, not universal. In real work, define practical equivalence using consequences, uncertainty, compute cost, interpretability, and maintenance requirements.
Inspect the Learning Curve
A learning curve compares training and validation scores as the training sample grows. A persistent large gap may indicate overfitting. Two low curves may indicate weak signal, poor features, or insufficient information.
Confusion matrix for the selected model on the untouched holdout set at threshold 0.50.
Report error counts alongside rates. With a small holdout, one changed prediction can substantially alter a metric.
Calculate Permutation Importance
Permutation importance measures how much holdout ROC AUC decreases when one input column is shuffled. It evaluates reliance on a feature in this fitted pipeline; it does not show causality, fairness, or the intrinsic value of that feature.
plot_importance = importance_table.sort_values("mean_importance").tail(12)fig, ax = plt.subplots(figsize=(8, 6))ax.barh( plot_importance["feature"], plot_importance["mean_importance"], xerr=plot_importance["sd_importance"], color="#2a7f62", alpha=0.85,)ax.axvline(0, color="black", linewidth=1)ax.set_xlabel("Decrease in holdout ROC AUC after permutation")ax.set_title("Permutation importance for selected pipeline")fig.tight_layout()fig.savefig(FIGURES_DIR /"permutation-importance.png", dpi=300, bbox_inches="tight")plt.show()
Permutation importance on the holdout set; error bars show one standard deviation across repeats.
Correlated and redundant features can divide importance. Negative importance on a small holdout can occur through sampling variation.
Perform Structured Error Analysis
Error analysis asks where the model failed and what those failures suggest about data, measurement, and use. It must not become a disguised search that repeatedly adapts the model to the holdout set.
Are errors concentrated in a service type, region, or missingness pattern?
Are wrong predictions confident or close to the threshold?
Could the target itself be inconsistently documented?
Is an omitted feature plausibly required at the prediction moment?
Would the operational consequence differ for false positives and false negatives?
Does any pattern need investigation in new data rather than another holdout-driven edit?
Do not publish record-level predictions or direct identifiers as portfolio evidence.
Compare Feature Sets as a Sensitivity Check
Evaluate whether engineered features improved the selected model family under cross-validation. This is a sensitivity check on development data, not a new reason to reopen holdout selection.
The saved object must include preprocessing and the fitted model. Saving only the estimator would require later users to recreate transformations exactly.
holdout_row = holdout_performance.iloc[0]model_report =f"""# Model Building Report: Follow-up Classification## PurposeEducational demonstration of feature engineering, pipeline tuning, nested validation, and model selection.## Development design- Development records: {len(development_data)}- Untouched holdout records: {len(holdout_data)}- Primary selection metric: ROC AUC- Inner validation: 4-fold stratified cross-validation- Outer validation: 5-fold stratified cross-validation## Selected specification- Model: {selected_model_name}- Hyperparameters: {final_searches[selected_model_name].best_params_}- Selection rule: {selection_reason}- Mean outer-fold ROC AUC: {selected_nested['mean_roc_auc']:.3f}- Outer-fold ROC AUC SD: {selected_nested['sd_roc_auc']:.3f}## Holdout results at threshold 0.50- ROC AUC: {holdout_row['roc_auc']:.3f}- Average precision: {holdout_row['average_precision']:.3f}- Brier score: {holdout_row['brier_score']:.3f}- Precision: {holdout_row['precision']:.3f}- Recall: {holdout_row['recall']:.3f}- F1: {holdout_row['f1']:.3f}## LimitationsSmall fictional dataset; uncertain target validity; internal random validation only; unstable fold and subgroup estimates; no external, temporal, or prospective validation; no impact assessment or monitoring.## Permitted useLearning, code review, and methodological discussion.## Prohibited useAllocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions."""(TABLES_DIR /"model-building-report.md").write_text(model_report, encoding="utf-8")print(model_report)
# Model Building Report: Follow-up Classification
## Purpose
Educational demonstration of feature engineering, pipeline tuning, nested validation, and model selection.
## Development design
- Development records: 555
- Untouched holdout records: 139
- Primary selection metric: ROC AUC
- Inner validation: 4-fold stratified cross-validation
- Outer validation: 5-fold stratified cross-validation
## Selected specification
- Model: logistic_regression
- Hyperparameters: {'model__C': 0.1, 'model__class_weight': 'balanced'}
- Selection rule: Random forest did not exceed the predeclared complexity margin; prefer logistic regression.
- Mean outer-fold ROC AUC: 0.679
- Outer-fold ROC AUC SD: 0.061
## Holdout results at threshold 0.50
- ROC AUC: 0.649
- Average precision: 0.592
- Brier score: 0.228
- Precision: 0.526
- Recall: 0.545
- F1: 0.536
## Limitations
Small fictional dataset; uncertain target validity; internal random validation only; unstable fold and subgroup estimates; no external, temporal, or prospective validation; no impact assessment or monitoring.
## Permitted use
Learning, code review, and methodological discussion.
## Prohibited use
Allocating or denying services, ranking people, automating follow-up, or making clinical, employment, financial, or eligibility decisions.
Interpretation and Limitations
Write a concise conclusion that answers four questions:
Did the selected procedure perform consistently across outer folds?
Did the more complex family clear the predeclared improvement margin?
What did the untouched holdout show about discrimination, calibration, and errors?
What evidence is still missing before any real-world use?
Important limitations include:
the data are fictional, small, and not representative of a defined population;
the target may reflect documentation or historical decisions rather than objective need;
derived features may encode operational practice rather than stable relationships;
nested cross-validation reduces selection bias but does not remove sampling uncertainty;
the random holdout does not test future periods, new sites, or changed workflows;
permutation importance is model-specific and non-causal;
error analysis on a small holdout is descriptive;
no model in this exercise is approved for deployment.
Export and Verify Results
Show code
expected_outputs = [ TABLES_DIR /"development-brief.csv", TABLES_DIR /"feature-plan.csv", TABLES_DIR /"feature-dictionary.csv", TABLES_DIR /"search-results.csv", TABLES_DIR /"nested-cv-results.csv", TABLES_DIR /"calibration-results.csv", TABLES_DIR /"holdout-performance.csv", TABLES_DIR /"holdout-predictions.csv", TABLES_DIR /"permutation-importance.csv", TABLES_DIR /"error-analysis.csv", TABLES_DIR /"model-selection-log.csv", TABLES_DIR /"model-building-report.md", FIGURES_DIR /"nested-cv-performance.png", FIGURES_DIR /"learning-curve.png", FIGURES_DIR /"calibration-curve.png", FIGURES_DIR /"holdout-confusion-matrix.png", FIGURES_DIR /"permutation-importance.png", MODELS_DIR /"selected-follow-up-pipeline.joblib",]output_check = pd.DataFrame({"path": [str(path) for path in expected_outputs],"exists": [path.exists() for path in expected_outputs],})assert output_check["exists"].all(), "One or more expected outputs were not created."output_check
path
exists
0
results/tables/development-brief.csv
True
1
results/tables/feature-plan.csv
True
2
results/tables/feature-dictionary.csv
True
3
results/tables/search-results.csv
True
4
results/tables/nested-cv-results.csv
True
5
results/tables/calibration-results.csv
True
6
results/tables/holdout-performance.csv
True
7
results/tables/holdout-predictions.csv
True
8
results/tables/permutation-importance.csv
True
9
results/tables/error-analysis.csv
True
10
results/tables/model-selection-log.csv
True
11
results/tables/model-building-report.md
True
12
results/figures/nested-cv-performance.png
True
13
results/figures/learning-curve.png
True
14
results/figures/calibration-curve.png
True
15
results/figures/holdout-confusion-matrix.png
True
16
results/figures/permutation-importance.png
True
17
results/models/selected-follow-up-pipeline.joblib
True
Render from the VS Code terminal:
quarto render
Open products/index.html in a browser. Check navigation, code folding, tables, figures, warnings, and whether every written claim matches the displayed evidence.
Portfolio Evidence
Your portfolio evidence should show the reasoning path, not only the winning model:
the model-development brief and prediction moment;
the feature-engineering plan and cautions;
the leakage-safe pipeline design;
the restricted hyperparameter search spaces;
nested cross-validation results with variability;
the predeclared complexity rule;
learning-curve and calibration interpretation;
the single holdout evaluation;
permutation importance and error analysis with limitations;
the saved pipeline, selection log, and model-building report;
a clear non-deployment boundary.
Project Question
Use this portfolio framing:
Can purposeful feature engineering and leakage-safe model tuning improve follow-up classification beyond the introductory specification, and is any improvement stable enough to justify additional complexity?
For another dataset, revise the unit, outcome, prediction moment, primary metric, candidate families, validation design, and complexity rule.
Workflow Summary
Summarize the completed project as a reproducible sequence:
restated the prediction task and intended-use boundary;
reproduced the Practical 004 cohort rules;
isolated a stratified holdout set;
planned and documented target-independent features;
constructed base and engineered feature sets;
placed learned preprocessing inside pipelines;
defined limited logistic-regression and random-forest search spaces;
used inner folds for tuning and outer folds for estimation;
applied a predeclared complexity margin;
examined learning curves and out-of-fold calibration;
evaluated the selected specification once on the holdout;
inspected errors and permutation importance cautiously;
saved, reloaded, and verified the complete pipeline;
exported a model-selection log and report.
Quality Checks
Document that:
cohort rules match the prior practical;
direct identifiers, contact fields, review status, and target are excluded from predictors;
engineered features are available at the prediction moment;
feature construction does not use target values;
the holdout was isolated before model comparison;
imputation, scaling, and encoding occur inside pipelines;
both candidate families use identical outer folds;
the inner loop tunes and the outer loop estimates;
the primary metric and complexity rule were defined before holdout evaluation;
probabilities, not class labels, were used for ROC AUC, average precision, Brier score, and log loss;
holdout evaluation occurred once after specification selection;
importance is described as predictive reliance, not causation;
record-level outputs are not published;
the saved pipeline reproduces predictions;
the rendered HTML has been inspected.
Reflection and Next Steps
Write brief responses to these prompts:
Which engineered feature had the clearest domain rationale?
Which feature was most vulnerable to encoding documentation practice?
How different were inner best scores from outer-fold scores?
Were the selected hyperparameters stable across outer folds?
Did the complex model clear the improvement margin?
What did the learning curve suggest about more data?
Were predicted probabilities adequately calibrated for the learning purpose?
Which holdout errors appeared most consequential?
Did permutation importance change how you understood the model?
What should be validated temporally or externally next?
Practical 006 can continue from this saved pipeline into model deployment: prediction interfaces, input validation, packaging, tests, monitoring, and safe release decisions.
Completion Checklist
Before marking the practical complete, confirm that you have:
Practical Complete
You have completed Data Science Practical 005: Model Building Practical.
You now have a reproducible model-development workflow that moves beyond fitting candidate algorithms. It connects purposeful feature design, leakage-safe tuning, nested performance estimation, calibration, model diagnostics, and an explicit selection rule.
Keep the unchanged input data, complete practical.qmd, exported evidence, saved pipeline, decision log, report, and rendered HTML together. They form a traceable record of what was tried, what was selected, what remained uncertain, and why a fitted model is not yet a deployed product.