A hands-on workflow for building a configured, validated, logged, and repeatable data pipeline with Python
Author
Complex Data Insights
Published
Aug 2026
COMPLEX DATA INSIGHTS
CDI Practicals
Hands-on exercises and step-by-step tools for doing, applying, and building with data.
Reproducible Data Pipelines Practical
Build a small pipeline that can be rerun, checked, explained, and handed to another analyst.
ID CDI-DATASCIENCE-PRACTICAL-007
Type Practical Product
Audience Students, researchers, analysts, professionals, mentors, and learners developing practical data science skills
Theme Reliable data workflows depend on reproducible pipelines that validate inputs, transform data consistently, record each run, and generate traceable outputs
Practical outcome
By the end, you will have a working pipeline that:
generates a deterministic synthetic input dataset;
reads behavior from a YAML configuration file;
validates schema, identifiers, categories, and numeric ranges;
creates analysis-ready data and a monthly summary;
publishes a reusable figure;
records a timestamped log and a JSON run manifest; and
stops immediately when a required condition fails.
Completion target: run the workflow successfully, inspect every output, force one controlled failure, repair it, and document the evidence that the rerun is reproducible.
Before you begin
You need Python 3.10 or later, a terminal, and a code editor. Work from the repository root so that all relative paths resolve consistently.
Create and activate a project-specific environment:
flowchart TD A[Configuration] --> C[Pipeline runner] B[Raw events] --> C C --> D{Validation} D -->|Pass| E[Clean data] E --> F[Summary and figure] F --> G[Log and manifest] D -->|Fail| H[Stop with error]
flowchart TD
A[Configuration] --> C[Pipeline runner]
B[Raw events] --> C
C --> D{Validation}
D -->|Pass| E[Clean data]
E --> F[Summary and figure]
F --> G[Log and manifest]
D -->|Fail| H[Stop with error]
The pipeline separates intent from implementation. Configuration says where inputs and outputs belong. Python code performs validation and transformation. Generated artifacts provide evidence of what ran.
Step 1: Inspect the repository contract
Review these locations before running anything:
config/pipeline.yml defines stable paths and allowed region values.
scripts/python/generate-example-data.py creates the raw input.
scripts/python/run-pipeline.py validates, transforms, summarizes, plots, and records metadata.
scripts/bash/run-workflow.sh runs the stages in the required order.
results/ contains tables, figures, logs, and quality-control evidence.
Write one sentence explaining why code, configuration, input data, and outputs should not be mixed in one folder.
Step 2: Read the configuration
Open config/pipeline.yml. Confirm that every path is relative to the repository root and that allowed categories are explicit.
Record two values that you could change without editing the Python pipeline:
Configuration value:
Effect of changing it:
Configuration value:
Effect of changing it:
Configuration improves reproducibility only when values are documented, version controlled, and read consistently. Moving every decision into configuration can make a small workflow harder to understand, so reserve it for values that vary across runs or environments.
Step 3: Generate the deterministic input
Run:
python scripts/python/generate-example-data.py
Inspect data/raw/service-events.csv. It should contain 480 records with these fields:
Field
Role
Expected rule
event_id
record identifier
unique and non-missing
event_date
event timestamp
valid date
region
grouping category
one of five allowed values
service_type
service category
descriptive category
wait_minutes
operational measure
zero or greater
satisfaction_score
outcome measure
between 1 and 5
The generator uses a fixed random seed. Running it again should recreate the same bytes in the raw CSV when the Python and dependency behavior remain compatible.
Step 4: Trace the validation gates
Open scripts/python/run-pipeline.py and locate the validate() function. Match each test to the risk it controls.
Validation
Risk controlled
Required columns
schema drift or incomplete export
Unique event_id
double counting
Allowed regions
spelling variation or unexpected category
Satisfaction range
invalid scale values
Non-negative waiting time
impossible operational value
Validation belongs before transformation. A polished chart produced from invalid data is still an invalid result.
Step 5: Run the complete workflow
Run both stages through the wrapper:
bash scripts/bash/run-workflow.sh
The terminal should report that the generator wrote 480 rows and that the pipeline completed successfully.
Confirm that the workflow created:
data/processed/service-events-clean.csv;
results/tables/monthly-service-summary.csv;
results/figures/monthly-service-volume.png;
results/quality-control/pipeline.log; and
results/quality-control/run-manifest.json.
Step 6: Check transformation logic
The processed dataset sorts events by date and identifier, then derives month. The summary groups by month and region and calculates:
number of service events;
mean waiting time; and
mean satisfaction.
Open the summary table and answer:
Number of summary rows:
First month represented:
Region with the most events in one selected month:
One check that confirms the counts are plausible:
A useful reconciliation check is that the sum of service_events across the full summary equals the number of rows in the clean dataset.
Step 7: Inspect the published figure
Open results/figures/monthly-service-volume.png. Verify that:
the title communicates the measure;
axes are labeled;
months are ordered chronologically;
the figure is readable without the source code; and
the plotted monthly totals agree with the summary table.
The figure is a pipeline output, not a manually edited illustration. If the source data change, rerun the workflow rather than editing the image.
Step 8: Use the log as execution evidence
Open results/quality-control/pipeline.log. The log records major stage transitions and the number of validated rows.
Answer:
What does the log prove?
What does it not prove?
A log confirms what the program reported during one run. It does not independently prove that the code was correct or that every relevant data-quality rule was tested.
Step 9: Read the run manifest
Open results/quality-control/run-manifest.json. Identify:
project name and status;
start and finish timestamps in UTC;
Python version;
input row count;
input checksum; and
checksums for the clean dataset, summary, and figure.
Checksums act as compact fingerprints. If file content changes, its checksum should change. Matching checksums provide strong evidence that two files contain the same bytes.
Step 10: Test idempotent reruns
Run the pipeline again without regenerating the input:
python scripts/python/run-pipeline.py
Compare the checksums of the clean dataset and summary before and after the rerun. They should remain stable. The figure checksum should also normally remain stable in the same environment. Timestamps in the manifest and log will change by design.
Idempotence means rerunning a stage with the same inputs and rules reaches the same intended output state. It does not require execution metadata such as timestamps to remain unchanged.
Step 11: Force a controlled failure
Make a temporary copy of the raw data. In the working raw CSV, change one region value to Unknown, then run:
python scripts/python/run-pipeline.py
The pipeline should stop with Unexpected region value detected. Restore the original input by rerunning the generator, then rerun the complete workflow.
Record the evidence:
Invalid value introduced:
Validation message observed:
Repair performed:
Final run status:
This exercise demonstrates fail-fast behavior: the workflow refuses to publish fresh outputs after detecting invalid input.
Step 12: Evaluate reproducibility boundaries
The project improves reproducibility, but it does not eliminate every source of variation. Consider:
dependency versions within allowed ranges;
operating system and plotting-library differences;
changes to source code or configuration;
external data sources that may be revised; and
manual steps performed outside the recorded workflow.
Choose one boundary and propose a stronger control.
Boundary:
Proposed control:
Why the control helps:
Examples include recording an exact environment lock file, storing immutable source snapshots, adding automated tests, or running the workflow in a container or continuous-integration service.
Step 13: Add one validation rule
Extend validate() with a rule for service_type. The accepted values are Consultation, Follow-up, and Screening.
Your rule should reject any unexpected value and return an informative error. Test it by introducing a temporary invalid category, then restore the generated data.
Step 14: Perform the completion checks
Mark each item only after inspecting the evidence.
Final reflection
The most important reproducibility control in this practical was:
The strongest evidence that my final pipeline succeeded was:
One improvement I would make before using this workflow in production is:
Practical summary
You built and tested a reproducible data pipeline with explicit configuration, deterministic input generation, validation gates, ordered transformations, documented outputs, logs, and file checksums. The central lesson is that reproducibility is not a single tool: it is a chain of controlled inputs, executable rules, inspectable outputs, and evidence that the workflow completed as intended.