The idea that good decision-making should be informed by the best available evidence is hardly a controversial one. Yet there remain many gaps in the evidence available to government.
Take crime reduction, for example. When it comes to organised crime, asset-focused interventions (AFIs) – such as confiscating property and recovering unpaid tax – are an increasingly popular law enforcement strategy. But while we know of around 300 studies on AFIs, none involves proper impact evaluation. This lack of evidence means that we have no certainty as to the effectiveness of AFIs in disrupting and deterring organised crime.
Improving the supply of evidence
We can – and must – do more to encourage universities and other research organisations to help us plug gaps in our knowledge. We have made a start. Since March, some departments have begun issuing Areas of Research Interest. These set out “the most important research questions” for each department that academic research could help address. First out of the blocks have been the departments of Health, for Transport, and for Environment, Food & Rural Affairs, as well as the Food Standards Agency. More will soon follow.
However, there are even bigger opportunities for departments to generate more of their own evidence on what works. What if policy teams routinely made policy in a fundamentally different way? What if their standard practice was to test variations in approach to a particular policy problem and then rigorously evaluate the results? What if it was business as usual for civil servants running frontline services to test new interventions before they are fully rolled out?
Harnessing our collective expertise
Experimentation is certainly not new to government. The Ministry of Agriculture and Fisheries (1919-1955) was a big sponsor of experimental methods in farming research, much of it at Rothamsted Experimental Station in Hertfordshire. Departments such as Work & Pensions and HMRC also have a long history of running controlled trials to discover what works. And the Behavioural Insights Team that was set up in the Cabinet Office in 2010 (and now operates as a social purpose company) spearheaded the use of randomised controlled trials (RCTs) in parts of Whitehall that had previously shown little enthusiasm for such activities.
But testing and evaluation is still hugely under-used across government. We need to strengthen our capacity to test new and varied approaches, and to build in rapid and robust evaluation. Only then can we shift resources from less to more effective programmes.
This is where the Cabinet Office What Works Team – supported by David Halpern as the National What Works Adviser – can help. It supports civil servants working in both policy and operational delivery roles to generate and use more evidence. It also strives to put in place institutional structures that improve the supply of evidence.
The team understands how complex it is to design trials and impact evaluations in government. And, at a time when budgets are constrained, it’s never been more important to share in-house expertise.
That’s why the What Works Team runs a cross-government Trial Advice Panel, bringing together some of the top trialling experts in the Civil Service. Panel members are on hand to offer free advice to policy teams on what sort of trial or test will generate the most useful results on what works. That might be an RCT, or equally it might be another type of experimental or quasi-experimental method. Panel members can also advise on evaluation design or be called on to offer guidance as challenges arise during a trial.
Thanks to support from the Economic and Social Research Council, the panel also consists of 30 UK-based academics from a range of disciplines, all of whom have first-hand experience of conducting high-quality trials and evaluations.
Since the panel was launched in 2015, members have assisted projects in 16 departments and public bodies. The team has seen plenty of examples – such as end-of-life care – where civil servants have been able to test new interventions in settings often dismissed as being too problematic for experimentation by government.
End-of-Life Support Trial, Cabinet Office
One of the earliest projects supported by the Trial Advice Panel was a Cabinet Office trial to test the evidence behind community-based end-of-life support – i.e. the use of volunteers to supplement state provision.
The Cabinet Office’s Centre for Social Action was keen to understand the impact of befriending services. For example, did they improve quality of life? Did they reduce the experience of loneliness and the burden felt by family members caring for their terminally ill relatives?
Trial-advice panel members offered advice concerning:
- eligibility criteria and consent
- when to consider the trial complete
- how to communicate with volunteer organisations (many of whom were not used to, or necessarily comfortable with, research trials in this area)
- how to liaise with the external organisations commissioned to evaluate the trial
Befriending services were found to have a positive impact on slowing the decline in participants’ physical health. The evaluation had important implications for the delivery of services. In terms of having a meaningful impact on quality of life, the trial pointed to targeting limited resources at certain groups who could receive more intensive support (e.g. older men who live alone), rather than spreading contact hours over a larger group of recipients.
The What Works Team also recognises that the key to stimulating greater interest in testing what works is to help civil servants understand experimental methods. With this in mind, it is collaborating with the Cross-Government Evaluation Group and the Policy Profession Support Unit to design and deliver training. The team is now involved in graduate Fast Stream inductions, the Future Leaders Scheme, and the development of new course material through Civil Service Learning.
Making use of the What Works Network
At the same time, it’s important that policymakers, commissioners and those delivering programmes and services have as much access as possible to the evidence that already exists. To assist in this, the team coordinates a network of seven independent What Works Centres – covering crime reduction, health, education, early years interventions, ageing, wellbeing, and local economic growth.
Nearly all of these receive direct government funding. However, they are not like other research institutions. They provide practical advice on the available evidence on different interventions – drawing on and interpreting evidence that is often highly technical in nature, buried in academic journals, or unpublished. Their staff members (or the partner organisations they work with) comb the internet and academic databases, systematically assessing the existing evidence, and offering accessible summaries for policymakers, practitioners, and commissioners.
For instance, the reviews undertaken by the College of Policing (who run the What Works Centre for Crime Reduction) suggest that CCTV, street lighting, and the intensive policing of crime ‘hot spots’ all reduce crime. On the other hand, there’s very little evidence that the electronic monitoring of offenders (through the use of ankle or wrist tags) has any overall effect on crime, despite being so widely used. Yet, in the case of one specific group – sex offenders – there is strong evidence to show that electronic monitoring reduces criminal activity.
Some of the centres also try to address gaps in the evidence base by commissioning their own trials. The Education Endowment Foundation (EEF) has had by far the biggest programme. The College of Policing is also supporting an increasing number of trials. Just recently, it has sponsored trials on stop-and-search and interventions to reduce domestic violence.
The Education Endowment Foundation’s RCTs
Almost one-third of all schools in England (a staggering 7,500 in all) have participated in trials funded by EEF. Nearly all of these were randomised control trials.
As a result, in just six years the EEF has more than doubled the amount of evidence we have from experimental trials in education in this country. When parents drop their children off at the school gates, they can now be assured that there’s a much stronger evidence base to support their child’s education.
We now know, for example, that the use of phonics is cost-effective in teaching young children to read (even more so if teachers receive formal training in phonics interventions). But older children who are struggling to read would likely benefit far more from other interventions, such as meta-cognition and reading comprehension strategies.
One of the biggest challenges remains the need to ensure that this evidence reaches decision-makers. This is critical if the What Works Network is to help practitioners improve the delivery of services and put decision-makers in a position to shift resources towards interventions that are achieving results.
A number of centres, including the College of Policing, have made their findings more accessible through a user-friendly toolkit. They appreciate that policymakers and frontline practitioners rarely have the time to locate and analyse evidence systematically. These toolkits allow users to compare the effectiveness and cost of different interventions, as well as the strength of the available evidence.
The centres also have outreach programmes to help embed their learning in practice. To take one example, the College of Policing recruits 'evidence champions' within police forces and runs a High Potential Development Scheme that promotes evidence-based policing.
The What Works Team supports the communication of the centres' findings through meetings and workshops across government, and via social media and publications. It also regularly brings the centres together so they can support each other as they develop their dissemination strategies.
The Team would like to work with as many other teams and units across government as possible to drive this initiative forward.
Why not:
- learn more about experimental methods by watching the Team’s online video;
- make use of the tools and resources produced by the What Works Centres; and
- consider using the Trial Advice Panel if you’re thinking of running a trial in your area of work.
For more information on the What Works initiative, email whatworks@cabinetoffice.gov.uk
For regular updates, follow the What Works Team on Twitter: @WhatWorksUK
Recent Comments