Description
Build a rigorous, leakage-safe model-development workflow with Python and scikit-learn.
This hands-on practical guides you through feature engineering, nested cross-validation, hyperparameter tuning, model selection, calibration assessment, holdout evaluation, structured error analysis, and evidence-aware reporting. You will work with cleaned synthetic service-delivery data and compare purposeful candidate pipelines while keeping the final holdout set untouched.
What you will learn
- Define a clear model-development brief, cohort, target, and prediction moment.
- Freeze an untouched holdout set before iterative model development.
- Engineer features reproducibly without introducing data leakage.
- Build mixed-type preprocessing and candidate pipelines with scikit-learn.
- Design purposeful hyperparameter search spaces.
- Use nested cross-validation for less biased model comparison.
- Assess performance, calibration, learning curves, feature importance, and error patterns.
- Apply a transparent model-selection rule and communicate responsible-use boundaries.
What you will produce
- A model-development brief, feature plan, and feature dictionary.
- Nested cross-validation and hyperparameter search results.
- Holdout performance, predictions, and confusion-matrix evidence.
- Calibration, learning-curve, and permutation-importance outputs.
- A structured error analysis and model-selection log.
- A saved end-to-end pipeline and model-building report.
Tools: Python, pandas, NumPy, scikit-learn, Matplotlib, seaborn, joblib, and Quarto.
Important: This practical is for education, portfolio evidence, and local technical testing. Its synthetic data and demonstration outputs are not validated for operational, clinical, or other real-world decisions affecting people.
