Programme overview
Introduction:
Impact evaluation of public programmes with RCTs, difference-in-differences, RDD and matching is a 2-week course for policy analysis, programme evaluation and evaluation commissioning teams that ends with an Impact Evaluation Design and Pre-Analysis Plan for a case programme. Many public programmes are scaled, cut or renewed on before-and-after comparisons that cannot separate the programme's effect from wider trends, self-selection and chance. Nominees already manage evaluations or analyse programme data, and the course is taught as a modelling build on simulated programme datasets, power calculations and cost data. CoreConcept Training Center delivers this impact evaluation of public programmes course.
Course Objectives:
- Convert a programme theory of change into causal evaluation questions with a defined treatment effect and counterfactual
- Select an identification strategy from the programme's assignment rules, eligibility thresholds and roll-out sequence
- Design randomised controlled trials, including cluster, phase-in and encouragement designs, with replicable randomisation
- Calculate sample size and minimum detectable effect and plan survey and administrative data collection for outcomes
- Estimate and interpret programme effects with difference-in-differences, regression discontinuity, matching and synthetic control
- Prepare cost-effectiveness estimates and a pre-analysis plan that inform budget allocation and scale-up decisions
Target Audience:
- Policy analysis units responsible for advising whether public programmes should be expanded, redesigned or stopped
- Programme evaluation teams responsible for designing and running counterfactual evaluations of public services
- Evaluation commissioning teams responsible for procuring, overseeing and quality-assuring external impact studies
- Funding and grant-making units responsible for allocating budgets on evidence of programme effectiveness
- Statistics and research units responsible for supplying survey and administrative data for programme evaluation
Course Outline:
Day 1: Causal Questions, Counterfactuals and the Purpose of Impact Evaluation
- Potential Outcomes Model and the Missing Counterfactual Problem
- Selection Bias and Confounding in Before-After Comparisons
- Theory of Change Converted Into Testable Causal Questions
- Average Treatment Effect, ATT and Intention-to-Treat Definitions
- Evaluability Assessment Checklist for a Public Programme
Day 2: Identification Strategies and Choosing an Evaluation Design
- Identification Strategy Decision Tree From Assignment Rule to Method
- Programme Operating Rules as Sources of Comparison Groups
- Internal Validity Threats Including Spillover and Contamination
- External Validity and Transferability of Estimated Programme Effects
- Evaluation Design Options Matrix for Phased Programme Roll-Out
Day 3: Randomised Controlled Trials for Public Programmes
- Individual Versus Cluster Randomisation Unit Selection Criteria
- Phase-In Randomised Roll-Out for Oversubscribed Programmes
- Encouragement Designs and Local Average Treatment Effect Estimation
- Stratified Randomisation Code That Is Verifiable and Replicable
- Attrition, Non-Compliance and Baseline Balance Table Diagnostics
Day 4: Sample Size, Statistical Power and Evaluation Data Collection
- Minimum Detectable Effect and Statistical Power Calculation Worksheet
- Intra-Cluster Correlation Adjustment for Clustered Sample Sizes
- Baseline and Endline Household Survey Instrument Design
- Administrative Records Linkage for Low-Cost Outcome Measurement
- Survey Data Quality Checks and Field Team Back-Checks
Day 5: Guided Case Study on Trial Design, Power and Estimation
- Conditional Cash Transfer Case File Randomisation Plan Review
- Case Power Calculation Under Budget and Cluster Constraints
- Case Balance Table and Attrition Report Interpretation
- Case Intention-to-Treat and Treatment-on-the-Treated Estimates
- Week-One Evaluation Design Memo Critique by Peer Panel
Day 6: Quasi-Experimental Designs for Programmes That Cannot Randomise
- Difference-in-Differences With Parallel Trends and Event-Study Checks
- Staggered Adoption Difference-in-Differences and Timing Bias Corrections
- Sharp and Fuzzy Regression Discontinuity at Eligibility Thresholds
- Propensity Score Matching With Common Support and Balance Tests
- Synthetic Control Method for Reforms Affecting One Unit
Day 7: Evaluation Ethics, Quality Assurance and Research Transparency
- Informed Consent and Ethical Review When Services Are Withheld
- Pre-Analysis Plan and Trial Registration Before Outcome Data
- Multiple Hypothesis Testing Corrections and Specification Searching Risks
- De-Identification and Secure Handling of Programme Microdata
- Replication Package and Independent Code Review Standards
Day 8: Commissioning, Stakeholder Engagement and Reporting Effect Sizes
- Impact Study Procurement Brief and Evaluator Independence Safeguards
- Evaluation Team Embedding With Programme Implementers From Design
- Effect Size Translation Into Plain-Language Policy Statements
- Confidence Intervals and Null Results Explained to Decision Makers
- Evaluation Policy Brief and Results Summary for Senior Officials
Day 9: Cost-Effectiveness, Heterogeneity and Evidence in Budget Decisions
- Cost-Effectiveness Ratio Calculation From Ingredients-Based Costing
- Cost-Benefit Comparison Using Monetised Programme Outcomes
- Heterogeneous Treatment Effects and Subgroup Analysis Discipline
- Meta-Analysis and Evidence Synthesis Across Comparable Evaluations
- Evidence-Informed Budget Allocation and Scale-Up Decision Criteria
Day 10: Modelling Build of the Impact Evaluation Design and Pre-Analysis Plan
- Case Programme Theory of Change and Causal Question Selection
- Case Identification Strategy Choice Among Randomised and Quasi-Experimental Options
- Case Sample Size and Minimum Detectable Effect Model
- Case Cost-Effectiveness Projection for the Budget Committee
- Impact Evaluation Design and Pre-Analysis Plan Completion and Defence
Skills You Will Gain:
- Counterfactual Reasoning
- Identification Strategy Selection
- Cluster Trial Design
- Statistical Power Analysis
- Quasi-Experimental Estimation
- Synthetic Control Analysis
- Programme Costing
- Evidence-Based Budget Advice
Why Attend This Course:
- Deliver an Impact Evaluation Design and Pre-Analysis Plan for a case programme to the head of the policy unit and the evaluation steering committee
- Decide which evaluation design a programme's operating rules can support before data collection starts and budgets are committed
- Avoid scaling or cutting programmes on before-after comparisons, underpowered samples or selectively reported results
- Pass on power calculation worksheets, balance table templates and pre-analysis plan checklists to evaluation colleagues
Conclusion:
Back at work, the participant presents the Impact Evaluation Design and Pre-Analysis Plan to the head of the policy unit, the evaluation steering committee and the budget team. They use it to decide how the next programme roll-out is assigned, how large the evaluation sample must be and which cost-effectiveness figure will justify scale-up or redesign. After the baseline survey, the unit should check achieved sample size, balance and attrition against the plan and revise the power assumptions and evaluation schedule.
Frequently Asked Questions (FAQ):
What should participants know before an impact evaluation of public programmes course?
Participants should already manage or analyse programme evaluations and be comfortable with averages, variance, regression output and hypothesis tests in a spreadsheet or statistical package. Bringing an anonymised programme description and its eligibility rules helps the capstone reflect a real roll-out.
How does an impact evaluation of public programmes course differ from a monitoring and evaluation frameworks course?
It estimates what a programme caused rather than tracking whether it delivered. Monitoring and evaluation framework courses build logframes, indicators and reporting systems, while this course builds counterfactual designs, power calculations and effect estimates with randomised trials, difference-in-differences, regression discontinuity and matching.
When should impact evaluation of public programmes use regression discontinuity instead of a randomised trial?
Regression discontinuity fits when a programme already assigns eligibility by a score threshold, such as a poverty index or test mark, and randomising is not acceptable. It estimates the effect for units near the cutoff, so results may not hold for units far from it.
What do participants take back from the impact evaluation of public programmes course?
Participants take back an Impact Evaluation Design and Pre-Analysis Plan for a case programme, with a causal question, a chosen identification strategy, a sample size and minimum detectable effect model, a data collection plan and a cost-effectiveness projection.