"If you can’t measure it, you can’t manage it."
Peter Drucker
Evaluation in the public sector can be a highly complex and challenging process. For an initiative as ambitious and far-reaching as the GREAT Britain campaign, this complexity is multiplied – yet GREAT’s willingness to confront its evaluation challenge head on has been at the heart of its success.
Launched in 2012, the GREAT Britain campaign is the Government’s most ambitious international marketing campaign ever. It showcases the very best the nation has to offer to encourage the world to visit, study and do business with the UK.
Operating under a single brand for all its activities, GREAT provides an integrated and consistent marketing platform for the international promotion efforts of the new Department for International Trade (DIT, formerly UK Trade & Investment), VisitBritain, the British Council and the Foreign & Commonwealth Office. It is the first time any country has managed to successfully unify its international promotional efforts in this way. Indeed, a total of 21 government departments and arm’s-length bodies (ALBs) now use the brand. It is active in 144 countries and 252 diplomatic posts worldwide and delivers approximately 80 to 100 economic activities and programmes each month (particularly in the key markets of China, the US, Brazil and India).
To maximise the impact of messaging to key target markets globally, GREAT leverages the existing strong brand recognition and support of iconic UK companies and high-profile individuals including British Airways, Jaguar Land Rover, the Premier League, Richard Branson, Andy Murray, Katherine Jenkins and the Royal Family.
Using sector-specific methodologies that have been validated by the National Audit Office (NAO), GREAT has already secured confirmed incremental economic returns of £2.7 billion for the UK, comprising:
- £1.77 billion from international and domestic tourism;
- £720 million from trade and foreign direct investment; and
- £228 million from international education.
Why did GREAT decide to place evaluation at the heart of its activities?
Implementing and managing a campaign of this nature – global, multi-partner and multi-sectoral – requires the constant and careful balancing of various, and often competing, priorities including brand development, project planning, securing funding, financial management, stakeholder management, team performance, competitor intelligence, market information and overall impact. To manage these effectively, GREAT took the strategic decision from the outset that evaluation had to be embedded in all aspects of the campaign to ensure consistency, evidence-based comparability and constant improvement.
This approach has proved highly successful – in 2015, a comprehensive appraisal by the NAO concluded by commending the campaign. This was followed by confirmation in the Spending Review that GREAT would receive government funding through to 2020, with an ambitious target of securing £1.6 billion per year of additional economic benefits for the UK. Today, given the number of government organisations involved with the campaign and the evaluation work they are each undertaking, GREAT may now be the most evaluated campaign across government.
So how does GREAT evaluate its activities?
Independent, conservative and best-practice evaluation techniques (based on the OASIS and GCS Evaluation frameworks) lie at the heart of the campaign.
To ensure that GREAT is meeting its target of obtaining a strong and evidenced-based economic return on (government) investment (ROI), the campaign is evaluated at three distinct levels:
- organisational level, where dedicated evaluation and monitoring teams in each delivery partner organisation use proven methodologies to track and analyse the incremental economic returns from their marketing activities;
- aggregated level, by the central GREAT team, comprising ongoing ROI analysis, the valuation of private sector support and overall campaign appraisal – importantly, all GREAT’s partner organisations are challenged robustly by the central team on an ongoing basis to ensure that their monitoring and evaluation methodologies are at a best-practice level, are providing the hard ROI evidence required for the campaign and, crucially, that additionality and attribution have been adequately accounted for; and
- overall governance level, through scrutiny by the campaign’s Senior Responsible Officer, review and sign-off from the ministerially led GREAT Programme Board and, finally, review by HM Treasury and the NAO.
Within this framework, the three key areas of sectoral analysis are:
- Tourism: VisitBritain’s industry-standard methodology for calculating ROI for its marketing activities is based on assessing the incremental uplift in visitor expenditure that is directly attributable to their intervention (see https://www.visitbritain.org/our-performance-reporting). Both the domestic and international GREAT tourism campaign evaluation results are typically available six to nine months after marketing activities.
- Education: When GREAT launched in 2012, the British Council had no specific methodology to calculate the incremental economic returns from their education marketing activities. To receive GREAT funding, the British Council developed a robust measurement approach from scratch, working closely with the central GREAT team and using best-practice evaluation approaches shared across the campaign by other delivery partners. Now, the British Council can clearly identify the incremental expenditure generated by international students choosing a UK university at both undergraduate and postgraduate level as a direct result of their interventions. At the heart of this is a comprehensive annual survey of international students at UK institutions, combined with a survey of prospective students in key overseas markets (such as China, India and the US). This innovative approach is providing the wider UK education sector with rich data on the full decision-making process of international students.
- Trade and Foreign Investment: Historically, DIT/UKTI has measured the impact of trade promotion using ‘service deliveries’ (essentially providing support to companies, ranging from attending an event to providing detailed export market reports and direct introductions to potential buyers). Following a recommendation by the NAO, this approach has gradually been enhanced, resulting in a new methodology that measures the ‘Additional Export Value’ created from DIT interventions with UK companies. This is adding even greater accuracy to measuring the real economic impact that DIT’s GREAT activities are having. Similarly, for foreign direct investment, DIT is implementing a new approach to calculate the additional economic value generated by international companies investing in the UK. These new methodologies will ensure that DIT is at the forefront of best practice in the measurement of trade and investment.
This comprehensive sectoral approach has enabled GREAT to implement, measure and then adjust its activities based on robust management information and evidenced results. Where activities have worked well, the campaign has re-invested. Where activities have performed less well, the campaign has been ruthless in reducing or removing investment and focusing on more effective areas (for example, the first year’s activities in Japan did not generate the expected level of outputs, so GREAT re-allocated funding in the following year to better-yielding markets such as China and the US).
To underpin the ROI evaluation work by GREAT’s delivery partners, the campaign also conducts regular international customer perception tracking surveys. These allow impact to be measured longitudinally across different sectors and geographies. Changing perceptions globally is a long-term process, so this ‘softer’ measurement is crucial – particularly given the focus on building relationships with global markets following the Brexit decision.
Finally, the GREAT central team has also been heavily involved in trialling and implementing new areas of measurement and evaluation, including the impact of soft power/return on influence, digital analysis and brand valuation (using private-sector best-practice standards). All of this innovative work keeps the campaign at the cutting edge of evaluation in the public sector.
What have we learned?
The diverse remit of the campaign has enabled GREAT to become involved in numerous evaluation challenges. The key recommendations we would make for others based on our experiences are:
- Build on existing knowledge. Ensure that you follow and use government best practice where it exists – there is no point reinventing the wheel.
- Ensure that evaluation is integral to all your activities before launching any new project. Also, make sure that the objectives you identify from the outset are all SMART (specific, measurable, achievable, relevant, timely).
- No matter how difficult, make sure that you evaluate the right long-term factors, such as actual project or organisational impacts (and not just simple activity outputs, such as clicks on websites or other early stage activity metrics).
- Evaluation is your friend – it helps you learn and improve and provides evidence of management effectiveness to stakeholders both internally and externally.
- Be objective. It often helps to have an independent perspective, so do not be afraid to bring in credible outsiders who can help to independently verify results.
- Invest in evaluation – this does not mean spending on expensive assessments or surveys, but it does mean that the full evaluation process is appropriately funded as part of the overall management of your project or campaign.
So, with apologies to Peter Drucker, perhaps a slightly more positive approach for the public sector would be… ‘if you can measure it, you can manage it’!
Recent Comments