We were the first to use propensity score matching to figure out the impact of the UK government’s innovation support on business performance.
We found robust evidence of positive effects on employment through careful analysis of administrative business data. Our experience has led us to continue to innovate in the use of large-scale datasets collected by private and public sector bodies to remain at the cutting edge in understanding the impact of public sector investment and policy decisions.
How much grunt for your grant?
The UK government invests almost £2 billion each year in business R&D – often in the form of grants to support particular innovation investments and research projects conducted by business. But there is surprisingly little robust evidence on the value for money of this spending. Does the spending make a difference to the firms that get the grants? Do they perform better, increase their revenues, or employ more people? And if so is this as a result of the investment or something else?
BEIS, Innovate UK and the National Measurement System were interested in understanding this so they could get a better handle on the value for money of grants for innovation support, and whether or not the effects depended on the type of investment or the type of firm receiving the investment.
Out of control
We used cutting-edge econometrics and big data to answer the client’s questions. We were provided with a detailed list of all the companies receiving innovation support from Innovate UK and the NMS over five years – this was our ‘treatment group’. We matched this with government administrative data on business performance.
We then used an econometric technique called “propensity score matching” to identify, among those that had not been supported, the firms which were most observably similar to the treatment group. This was our ‘control group’. We tracked outcomes for the two groups up to five years after getting the support to see what happened and compared the trends for the two groups. At the time, nobody else had used this method to look at the impact of innovation support on business performance.
We worked closely with one of the leading academic practitioners of the approach to ensure that what we did was robust, leading-edge and intellectually sound.
Surviving, then thriving
We found that receiving support made it much more likely that firms survive, and strong evidence of positive effects on employment. We also found that turnover increased, but the results were a bit less robust.
As we were the first to use these methods in this context we uncovered some methodological challenges that have improved our and our clients’ understanding of the strengths and limits of these novel approaches to estimating the impact of policies on businesses. For example, such methods work less well for very large firms where no reliable ‘control firm’ exists.
This means that a holistic picture of the impact of these large-scale funding programmes requires complementary methods including case studies and bespoke data collection. Combining these approaches, informed by a clear understanding of the logic of an intervention, underpins ‘theory-based’ evaluation, increasingly applied to large-scale programme evaluation where Frontier has been at the forefront of thinking in recent years.