Training Course

Policy Evaluation methods course

Speaker

Barbara Sianesi

Date & Time

From: 28 March 2022, 10:00
Until: 29 March 2022, 17:30

Type

Training Course

Venue

UCL Economics Department
Drayton House,
30 Gordon Street,
London,
WC1H OAX

Prices

HE Delegates: £504
Charity or Government: £924
Other Delegates: £1994.40

Please note the dates for this course: 7 – 27 March 2022 for self-paced study, March 28 and 29, 2022 for the 2 live days.

Prices include VAT.

TRAINER’S BIO

Please click here for CV.

FORMAT OF THE COURSE

The course consists of a mix between self-paced study and live in-person meetings:

PART 1) 21.5 hours of pre-recorded lectures. Participants will receive access to the lecture videos for 6 weeks (3 weeks prior to and 3 weeks after the live classes) to go through them at their own pace. It is highly recommended that participants have finished watching the lectures before the live meetings, which are devoted to questions and Stata practicals. Extended access to the videos after the end of the course is meant to be used for a second listening in order to revise/review/deepen/refresh specific (or all) topics.

  • Introduction to the Course
  • Research Design, the Evaluation Problem & overview of Analytical Challenges
  • Evaluation Methods
    • The Naive Estimator
    • Randomised Experiments
    • Instrumental Variables
    • Regression Discontinuity Design: Sharp
    • Regression Discontinuity Design: Fuzzy
    • Regression Methods
    • Matching Methods
    • Longitudinal Methods: Before-After
    • Longitudinal Methods: Difference-in-Differences
    • Longitudinal Methods: Synthetic Control Method
  • Wrap-up & Conclusions

PART 2) 2 days of in-person classes consisting of

  • A quick summary/recap of each topic, taking questions relating to that topic
  • Time for participants to work on the corresponding guided Stata practical
  • Barbara going through and discussing that Stata practical live

A rough breakdown of the live sessions:

 Day 1 – Monday 28 March 2022Day 2 – Tuesday 29 March 2022
9.30Registration and coffeeCoffee
10.00Evaluation Problem Naive EstimatorRegression Discontinuity Design: Fuzzy Regression Methods
11.30CoffeeCoffee
11.45Randomised ExperimentsMatching Methods
13.15LunchLunch
14.00Instrumental VariablesBefore-After Difference-in-Differences
15.30TeaTea
15.45Regression Discontinuity Design: SharpSynthetic Control Method
17.30End of Day 1End of Day 2

COURSE DESCRIPTION

How can one evaluate whether a government labour market programme such as the Work Programme, or a subsidy to education such as the EMA is actually working? This course deals with the econometric and statistical tools that have been developed to estimate the causal impact on one or more outcomes of interest of any generic ‘treatment’ – from government programmes, policies or reforms, to the returns to education, the impact of unionism on wages, or of smoking on own and children’s health.

After highlighting the ‘evaluation problem’ and the challenges it poses to the analyst, we focus on the main empirical methods to solve it:

  • Naive non-experimental estimator
  • Randomised social experiments
  • Natural experiments or instrumental variables
  • Regression Discontinuity Design
  • Regression analysis
  • Matching methods
  • Before-after
  • Difference-in-differences
  • Synthetic control methods

For each of these approaches, we give the basic intuition, discuss the assumptions needed for its validity, highlight the question it answers and formally show identification of the parameter of interest. The relative strengths and weaknesses of each approach are discussed in detail, drawing from example applications in the economics literature.

Each method will be implemented ‘hands-on’ in practical Stata sessions.

By the end of the course, participants will be able to:

  • frame a variety of micro-econometric problems into the evaluation framework, and be aware of the concomitant methodological and modelling issues;
  • be discerning users of econometric output – able to interpret the results of applied work in the evaluation literature and to assess its strengths and limitations;
  • access the evaluation literature to further deepen knowledge on their own;
  • choose the appropriate evaluation method and strategy to estimate causal effects in different contexts; and
  • use simple statistical packages (we use Stata in the course) to implement the different evaluation methods to real data.

LEVEL OF KNOWLEDGE REQUIRED

This is an intermediate-level course on quantitative empirical methods for policy evaluation. As such, familiarity with basic statistical concepts (e.g. significance testing) and basic econometric tools like OLS regression and probit/logit models is required. Note also that there is a certain degree of formalisation (at the level of this paper) – though this part is optional. Finally please note that while offering an in-depth and thorough overview and discussion of the various evaluation methods, this is not an advanced course at the post-graduate level. Most emphasis is devoted to understanding the issues, to the choice of the most appropriate method for a given context and to the implementation of evaluation methods in practice.

The practical part of the course will make use of Stata; although the exercises will be guided, basic familiarity with this software is strongly recommended.

Note also that while participants have often come from a range of backgrounds and have reported enjoying the course, the examples and discussions come from a mostly (labour) economics perspective.

BACKGROUND READING

No preparatory reading is required as the course does not assume prior knowledge of evaluation methods and issues.

Some past participants have however found it useful to scan the following paper in advance of the course to get an idea of the issues, notation and more generally some of the material:

Blundell, R., Dearden, L. and Sianesi, B. (2005), “Evaluating the Effect of Education: Models, Methods and Results from the National Child Development Survey”, Journal of the Royal Statistical Society, Series A, 168, 473-512. IFS Working Paper.

Alternatively, a gentle introduction to evaluation issues and methods is offered by Ravallion, M. (1999), “The Mystery of the Vanishing Benefits: Ms. Speedy Analyst’s Introduction to Evaluation”, World Bank Policy Research Working Paper No. 2153


PAST PARTICIPANTS’ FEEDBACK

Public sector (January 2021): Brilliant course. Lectures fantastically presented and explained. Barbara is clear in her explanations and explains things in multiple ways, as well as explaining the intuition well. Online pre-recorded lectures done very well – use of slides and annotations done well. Recording also done really well. The pre-recordings gave me time to really think about things deeply before the live sessions.I really like the split of pre-recorded sessions and live sessions. Overall fantastic! Barbara is super engaging and explains this so well. I thought there was a great level of detail while still being very accessible since Barbara starts with the basics and explains things informally as well as formally. Really good use of slides/annotations etc when presenting virtually despite this being the first time the course was run online! It doesn’t seem that way! Thanks a lot 🙂

Private sector or other (January 2021): The amount of material covered in this course is very impressive, the lectures are clear and well explained and making them compulsory prior to the live course is very important.

HE lecturer or other academic (January 2021): Barbara’s excellent – very patient with questions, very professional, and gives very helpful advice.

HE lecturer or other academic (January 2021): Just to say thank you Barbara, this was a great course and there was something in it for all levels of learner

HE lecturer or other academic (March 2021): This is an excellent course! Extremely helpful for my own research. Watching the recorded lectures are particularly useful for me to have a good understanding of the lectures, and then the live sessions are ideal for getting my questions answered.

HE student (March 2021): I have no comments other than thank you for the very clear explanations and plenty of examples! This was a great course for participants of all levels!

HE student (March 2021): This is a great course – it was recommended to me by several of my colleagues and I’ll keep recommending it to people in turn.

HE lecturer/academic (May 2021): Barbara is an excellent teacher, explaining everything well with authentic examples. An excellent course! Many approaches, done thoroughly.

HE student (May 2021): Very interactive. Barbara did a great job with the format and content of the sessions. The content is intensive but she simplified the material in an easy to follow and understand manner. Also the format to do a topic and then the practical exercise was good. I also like that the lecture slides and vidoes were shared in advance for the self paced learning before the live session. It was also great to have flexibility whether to join a breakout room or not. Excellent course and excellently delivered by Barbara.