home | strategy development | strategy skills | site index | access keys

Strategy Skills > Building an Evidence Base

Learning from others - International comparisons

> in practice

International comparisons bring together information, data and analysis on approaches to tackling similar policy areas in other countries and draw out relevant experience that may be applied in the UK. They can be one of the only real ways of comparing effectiveness of different strategies aimed at the same or similar problems (if in different institutional and cultural contexts). Benchmarking of different countries can also provide pointers to how to improve UK performance across a wide range of issues - by pointing to countries which are leading the field (identifying positive deviance from the norm).

Often conducted as a stand-alone piece of work, the output is usually a written report but could also be a presentation or seminar. It can also feed into the overall project report.

The Process

1. Identify problem or challenge

In the early stages of analysis, it is often helpful to define the problem being considered fairly flexibly or broadly and without couching it in institution-specific terms that might inhibit cross-national comparison. Once potential comparator countries have been identified, it will be possible to focus on the particular issues where international comparisons might be most instructive. The key questions to be addressed and the information to be acquired from an international mapping exercise should be carefully defined. A pro forma can be a helpful tool, particularly to guide web-based searches.

2. Identify comparator country/region

It is often useful to survey a wide variety of countries to see which might be the best 'fit'. The ideal comparator is one where there is a similar policy environment i.e. where the issues faced are closely comparable, and also where the two countries are similar in cultural and socio-economic respects. Of course, two policy environments will never be identical, and it is a matter of judgement as to the extent to which differences in ideological, resource and institutional factors at either the macro or micro level impact on the ability to make worthwhile comparisons.

3. Gather relevant evidence

In gathering evidence to inform cross-border learning, it is advisable to consult as wide a range of relevant sources as possible. From a distance, it can be difficult to map out the inter-relationships between different stakeholders, and determine the significance of different elements of the institutional landscape. Gaining a wide variety of perspectives on an issue helps to fill in essential contextual information and provide a more rounded understanding.

In gathering relevant evidence, a key decision is whether to visit the country in question or to collect information at a distance. If you are planning to conduct country visits, sufficient time and budget should be allocated in the project management plan. For distance data gathering, the internet is of course the most useful resource, in addition to telephone calls, correspondence and video conferencing.

Consider meeting, or corresponding with, the following: policy-makers/decision-makers; programme sponsors, managers, staff and other stakeholders (where a specific government programme or policy is being considered); target participants; evaluation and research/academic community, and representatives from the media.

Other sources of information include international bodies (e.g. European Commission, OECD, UN, World Bank, IMF) and Universities which have comparative research expertise in the area in question.

Most Departments will have good international contacts and should be able to point you to the relevant institution or person. The Foreign Office (FCO) will also be able to provide contacts in the relevant countries. However, the FCO receives a large number of requests for assistance from different departments. It is therefore important to use the FCO only when essential and to be very specific as to the information required. For example it may be possible to obtain the information required from other projects or ongoing work. If it is necessary to go through the FCO, see below for guidance on working with embassies.

The following points should be noted when considering international trips

• The value of a trip can be greatly reduced if insufficient research is carried out in preparation beforehand.
• Difficulty in identifying people to meet/key players - this can be far harder than it might appear - ask as widely as possible, both here and in the country you are planning to visit. It is far preferable to have too many contacts than too few! Also important to identify what information and data you would like to obtain from each of the individuals you meet based on their specific areas of expertise.
• Actual practicalities of undertaking a trip - in preparation and for the trip itself can both be very time consuming and tiring. Plan well in advance and be realistic about the number of meetings able to be accommodated on a trip.
• Follow-up after a trip - making sure lessons are learnt are integrated into mainstream work, including disseminating them effectively to colleagues. This can be difficult because some information/lessons will not be needed until much further down the line. Recording all information (writing notes of all meetings during the visit), keeping good contact lists and filing all papers is essential - it is surprisingly easy to forget the detail once you are no longer immersed in it!

4. Interpret relevant evidence

When interpreting international evidence it is important to bear in mind that whilst international experience can serve to inspire new lines of enquiry or, in some cases, constitute strong evidence that a particular policy idea is likely to be succeed or fail, it cannot itself give us the answers - potential solutions also need testing in the domestic context.

Tips for Undertaking International Comparisons

• Building in international comparisons at the most useful stage in the project - you need to have developed your own thinking sufficiently in order to ask detailed and focussed questions (especially if you are visiting in person) but it also needs to be early enough in the project to allow comparative international experience to shape your subsequent thinking
• Need to specify carefully a limited number of countries (around 6) and what is required in the comparison; context is very important.
• The team should be prepared to take the comparison work forward - academics (if you ask one to do the study) are often not best placed to draw out the issues, gaps and implications for the UK.
• Difficulties in getting anything other than anecdotal evidence even from 'experts'. The quality of evidence available internationally may be fairly patchy, especially when there is a lack of comparability in data sets etc between countries. Lack of familiarity with a different policy setting, and lack of time available to devote to international comparisons, makes unpicking research carried out overseas more difficult than analysing the findings of UK research.

Working through UK embassies abroad

Staff at UK embassies abroad can be incredibly helpful in providing background information, finding appropriate contacts, setting up meetings and providing cultural commentary on emerging conclusions. However, each embassy individual has to cover a wide portfolio and is having to juggle a wide range of requests.

The FCO has produced best practice advice for working with posts in EU capitals, which is also appropriate for working with posts world-wide:

• Explain the background to the requests/instructions.
• Write clearly, and if your note is to be handed over, write for a non-English speaker.
• Identify which posts you need to approach.
• Allow time.
• Set out our position/thinking, and highlight key points
• Include a speaking note where possible.
• Put a contact name, telephone number and e-mail address at the bottom of your request, and do not send your email to a random name in the post.
• When in doubt, check with the FCO and seek our advice. Always keep us aware of what you are doing.
• Please provide feedback to posts on how helpful the information has been and copy them the results. It enables them to respond more effectively next time.

Other best practice tips include:

• The structure of embassies varies, as do titles/portfolios for individual posts, so a little research is required to ensure you are contacting the correct person. It is probably best to start by contacting the relevant FCO country desk officer in London, who can then direct you to the appropriate contact in Post.
• The staffing levels of embassies varies greatly and needs to be taken into account when commissioning work. Select which countries you really need examples from, consult FCO on likely embassy capacity and ask posts to highlight any particular local issues.
• When commissioning work, it is best to provide details of background sources, with web addresses where possible. Embassy contacts are usually not specialists and this helps them get up to speed on the issue.
• Always provide details of the situation in the UK. This provides information for the embassy contact to trade with local officials. Also, if you have asked for a return from the embassy, the UK details will provide a template indicating desired coverage and level of detail.
• Initial requests should be copied to the relevant geographical department in the FCO, as well as any subject-based FCO contact(s). This allows FCO to keep track of the non-FCO requests being put to Embassies.
• The FCO are developing a 'Science and Technology' network, with a UK-based hub which provides advice and can act as a conduit for requests. Other networks being established include the Environment network and the Energy network.
• Departments will have on-going contacts with embassies and sometimes have specialists in policy interactions with particular markets. Ensure that you talk to all UK-based contacts in parallel with posts.
• Ask Departmental and Embassy contacts about any in/formal bilaterals or conferences in your subject area. It may be possible to attend or ask that specific questions be raised on your behalf.
• If it is likely that other international comparisons are to be requested in the future, it is worth sending a warning to the relevant embassies. This will disappear in the maelstrom of paperwork in some posts, but others will start thinking and collecting relevant information on what may not be a specialist topic.
• Be aware that holiday seasons vary between countries. For example, Sweden takes its eight week summer holiday from mid-June to mid-August, and so arranging meetings in July can be difficult. UKREP contacts in Brussels recommend avoiding the first month of presidencies where possible.

Strengths

• Provides a real insight into strategy development and context in other countries.
• Provides a framework for assessing UK performance and strategy/policy gaps.
• Provides ideas to pursue in the policy development stage of a project.
• International comparisons are best used when the issue being addressed is very clear-cut (the regulation of simple monopolies for example). They are least useful where important underlying circumstances are radically different.

Weaknesses

• It's easy to get bogged down in irrelevant details whilst trying to get to grips with a new policy setting - the trick is to isolate and focus on the most relevant facts.
• Having too many objectives and too wide a range of evidence you're looking for - important to clearly define and focus your enquiry.

Useful Sources of International Comparative Data

Reform Monitor. This site tries to keep up to date with different government reforms instituted in the areas of social policy (health care, pensions provision, family policy, state welfare), labour market policy and industrial relations. It is international in scope (15 OECD-countries: Australia, Austria, Canada, Denmark, Finland, France, Germany, Italy, Japan, Netherlands, Spain, Sweden, Switzerland, United Kingdom and United States of America). It doesn't seem to keep completely up to date (certainly not in relation to UK policy) but provides some useful descriptive comparisons of policies in OECD countries.

Campbell Collaboration. Newly constituted international collaboration which aims to assess the actual effect of social and educational reforms more rigorously than has been done in the past. It is closely linked with its sister organisation, the Cochrane Collaboration, which aims to do the same assessment of "what works" for medical interventions.

Economist Country Briefings. The Economist has made many of its country-specific articles and surveys available on its website. These can provide useful context and background material.

Public Management OECD Country Information. The OECD maintains a country-by-country resource on developments in governance and public management.

International Comparisons toolkit on the Policy Hub

References

Almost every Strategy Unit project has undertaken international comparisons and these are worth a look, often to be found as annexes in reports. For example, see the Adoption Review (July 2000) and the Ethnic Minorities in the Labour Market report (Feb 2002).

There is a wealth of academic literature on comparative political science (focussing on the viability of making cross-national comparisons) and specifically on policy transfer ('exporting' policies from one setting to another).

Global Comparisons in Policy-Making: the view from the Centre, Geoff Mulgan, June 2003.

^ top