How best to learn the lessons from policy interventions

Monitoring and evaluation is an essential part of informed decision-making.

Evidence generated from monitoring and evaluation can help policymakers to understand what works, for whom and under which circumstances. The lessons learned are then used to strengthen future policy design and appraisal methods.

But what is the best approach to assess the impacts of a policy or intervention? The answer is not straightforward, as the transport sector demonstrates.

The past few years have been fascinating for the sector. The Covid-19 pandemic and subsequent lockdowns had a dramatic impact on transport use. Although passenger numbers are recovering, there are signs that people are making different choices about work, education, and social and recreational travel. The behaviours of the past may not be a good indication of the behaviours of today.

Alongside these behavioural changes, there have been advances in technology, a big push to decarbonise and major funding allocations to improve transport and support local communities.

The relevant schemes include the City Region Sustainable Transport Settlements; Bus Service Improvement Plans; Levelling Up Fund; Local Transport Fund; Towns Fund; and the temporary £2 bus fare cap, which aims to save passengers money and encourage more people back on the bus.

This landscape poses evaluation challenges. Two stand out:

Availability of comparators

Many quantitative impact evaluation methods rely on determining a counterfactual, i.e. what would have happened in the absence of the intervention. This is typically established either by using historical trends or identifying a control group.

Both these options have become increasingly problematic. The impact of Covid-19 means that recent historical trends are unlikely to be representative of future travel habits, while the wide range of policy interventions means it is rare to be able to single out an area that is not also the recipient of significant transport or local community investment.

Understanding the impacts of particular projects

There are many parts of the country where several policy interventions are currently taking place in parallel, with potential inter-dependencies between them. For example, some areas have built on the national £2 bus fare scheme to offer additional discounts using BSIP funding.

An integrative approach to evaluation helps to overcome the resulting analytical challenges. It enhances our ability to draw conclusions by bringing together various methods that view the assessment through different lenses.

This can be done, for example, by integrating theory-based and quasi-experimental methods, which both generates a wider range of lessons and helps to increase the reliability of findings:

While quasi-experimental methods are useful to understand the average impact of an intervention, they are not so good at pinning down why the impact occurred, which subgroups were affected, and under which circumstances.

These questions are often better addressed using theory-based methods. Theory-based approaches such as realist and contribution analysis can be employed to triangulate findings from quasi-experimental data, particularly where the findings may be tentative due to data limitations.

The following case study of the £2 bus fare scheme illustrates the integrative approach to evaluation.

Impact evaluation toolbox

Theory of change

A theory of change sets out how an intervention is expected to work (casual chains), the assumptions made, the quality and strength of evidence supporting these assumptions, and the wider contextual factors. It maps how inputs and activities are expected to deliver outputs and ultimately outcomes targeted by the intervention.

Theory-based

Theory-based methods draw conclusions about the impact of an intervention through testing the casual chains in the theory of change.

Quasi-experimental

These methods are used to quantify the impact of an intervention by comparing observed outcomes with a counterfactual, i.e. what would have happened without the intervention.