The Policy and Organisational Context for Commissioned Research

Return to front page of this collection of resources

Contents:

1. Explanation of some common terms
2. Modern policy-making and the need for knowledge;
3. The diversification of ‘knowledge’;
4. ‘Evidence-based policy’: the proliferation of ‘policy’ and why ‘evidence’ remains a challenge;
5. Research and policy: Marriage, divorce or just good friends?;
6. Knowledge, skills, values and attitudes: how researcher identity is constructed in policy environments;
7. Further reading and bibliography.

This resource provides an overview of the policy context for commissioned research/evaluation in the public sector, including: an explanation of some common terms; modern policy-making and the need for knowledge; how ‘knowledge’ has been diversified; ‘evidence-based policy’ and why it remains a challenge; the current state of the relationship between policy and research; and the construction of researcher identity in policy environments.

1. Explanation of some common terms

Policy, policy-making may appear to be self-evident terms, though they are often conflated with ‘politics’ and need some elucidation. ‘Policy’ may be used in a general or a specific sense (e.g. ‘the department’s whole policy on conflict resolution was vague’ = espoused position; ‘the governors had just re-drafted the policy on the use of mobile phones in school’ = written guidelines). A policy may be manifested in a variety of ways through, for example, models, initiatives, programmes, projects, services, products, events.

Wikipedia gives this definition of policy:

… a policy is a deliberate plan of action to guide decisions and achieve rational outcome(s). The term may apply to government, private sector organisations and groups, and individuals… Policy… may also refer to the process of making important organisational decisions, including the identification of different alternatives such as programs or spending priorities, and choosing among them on the basis of the impact they will have. Policies can be understood as political, management, financial and administrative mechanisms arranged to reach specific goals. (http://en.wikipedia.org/wiki/Policy_making accessed 19 January 2009.)

Rather more grandly, the New Labour Government defined policy-making thus:

‘the process by which governments translate their political vision into programmes and actions to deliver “outcomes” – desired changes in the real world’. (Modernising Government White Paper, 1999 )

This resource is more concerned with policy-making than policy itsel.

Policy-maker is a term customarily used rather loosely, including in this resource, referring variously and depending on context to elected politicians (at parliamentary/assembly level and in local government) and most especially front-bench ministers charged with fulfilling manifesto commitments; senior civil servants and advisers in central government with responsibility for formulating and advising on policy design and strategy, and specific initiatives; local authority officers and other professional staff with similar remits in the public and voluntary sectors – not to mention governing bodies of schools and the like. The term can also be stretched to cover organisations whose remit requires them to exert influence over national policy – the regulatory bodies, the teachers’ and headteachers’ professional associations, charitable foundations, campaigning organisations and so forth. This extremely loose usage is no doubt partly due to the extended and dispersed nature of modern policy-making processes, as well as to the generic character of the word ‘policy’ itself. Nonetheless, it is likely that the commonest interpretation of the term outside of government is ‘ministerial politician’ – which may have the effect of obscuring the role of the civil service/professional officers in guiding, challenging, tempering and, crucially, providing evidence in favour of or against particular courses of action.

The national climate in which educational research and evaluation are commissioned has been significantly influenced by the weather-front of evidence-based policy (or evidence-based education, EBE), which has blown in from the west, along with other education policy borrowings from the USA. Simply put, this is the avowed intention that decision-making in the education system be more effective and therefore founded upon ‘the best available’ evidence rather than upon ideology; it also serves the function of strengthening the legitimacy of policy decisions. The fundamental problem with the ‘evidence-based agenda’ is that policy will always take (prior) account of factors like public opinion, manifesto commitments, the economic climate and international developments – leaving aside any difficulties such as the availability/quality of the evidence.

Research is another rather loosely-used term, which is used in different contexts to mean the process of inquiring into a question or a problem, the result or outcome of such an inquiry (in the form of theories and/or findings – ‘knowledge’), a discipline surrounded and supported by an apparatus of peer-reviewed publication, etc., or to point towards a set of values concerned with ideas of truth and impartiality. Again, this resource makes free with all these definitions and associations.

Although policy-makers take notice of research findings originated by researchers (e.g. through grant-funding), they are probably more likely to use internally-commissioned research. Contract research is research commissioned and procured by an organisation in the private or public sector to meet some policy need – for information, evidence, confirmation, etc. It is set out within defined parameters of time, budget, objectives and sometimes design. These parameters constitute the main components of the contract, which is a legally-binding agreement between the agency and the research provider (whether individual or institutional). It is customary to refer to the commissioning agency (and/or the representative thereof) as the client or customer, to whom the researcher as contractor is providing a specified service; this terminology itself indicates that the relationship is a much more direct, and directive, one than that between researchers and grant-givers or funders. See Resource 2 for more on this.

Much, though by no means all, of the research that is commissioned by public (or private) bodies is an evaluation of a particular policy, usually of the extent to which the policy is meeting or has met its aims, together with, for example, an assessment of its impact on different stakeholder groups, key success factors, major barriers and risk factors, any unintended consequences and possible recommendations for future actions. Evaluations are consequently not open-ended, especially in terms of outcomes, in the way that research projects may be, but they make extensive use of research methods and instruments – which is why the terms are often bracketed together, or even used interchangeably.

2. Modern policy-making and the need for knowledge

One of the interesting challenges in these resources is the fact that literature on policy(-making) in education tends to be polarised into two kinds: policy statements, which are about intentions, plans, objectives and political rhetoric; and academic commentaries on policy, which are often theoretical, abstract, critical, partial. In order to understand the role of research in policy, we need analytical ‘thick’ descriptions of the policy-making process – including whether, when and how the need for ‘evidence’ is expressed and acted upon, though these are rather less common (see Moss and Huxford 2007, Somekh 2007 for two recent examples).

Useful information about policy-making can be gleaned from the way it is perceived by those charged with developing requisite skills and competences to support the policy process; they include the Policy Hub, an on-line resource created by the Government Social Research Unit; the Magenta Book guidance on policy evaluation; and the National School of Government, which provides professional training and development for staff working in national and local government.

According to the Cabinet Office document ‘Professional Policy Making for the Twenty First Century’ (published in 2000), ‘modernised’ policy-making is:

• Strategic – looks ahead and contributes to long-term government goals;
• Outcome focused – aims to deliver desired changes in the real world;
• Joined up (if necessary) – works across organisational boundaries;
• Inclusive – is fair and takes account of the interests of all;
• Flexible and innovative – tackles causes, not symptoms and is not afraid of experimentation;
• Robust – stands the test of time and works in practice from the start.

The same documents identified the features of an effective policy-making process as follows:

‘A policy-making process which is fully effective:

• clearly defines outcomes and takes a long term view, taking into account the likely effect and impact of the policy in the future five to ten years and beyond;
• takes full account of the national, European and international situation;
• takes a holistic view looking beyond institutional boundaries to the government’s strategic objectives;
• is flexiible and innovative, willing to question established ways of dealing with things and encourage new and creative ideas;
• uses the best available evidence from a wide range of sources; (our emphasis)
• constantly reviews existing policy to ensure it is really dealing with problems it was designed to solve without having unintended detrimental effects elsewhere;
• is fair to all people directly or indirectly affected by it and takes account of its impact more generally;
• involves all key stakehoders at an early stage and throughout its development;
• learns from experience of what works and what doesn’t through systematic evaluation.’ (ibid.)

These kinds of framework, good practice and guidance documents give insights into what the ideal policy process is; and all current and recent such documents give a high priority to evidence, evaluation and research – often expressed as ‘high quality information’ – at all stages of the process from formulating particular policies:

‘Good quality policy making depends on high quality information, derived from a number of sources – expert knowledge; existing domestic and international research; existing statistics; stakeholder consultation; evaluation of previous policies; new research, if appropriate; or secondary resources, including the internet. Evidence can also include analysis of the outcome of consultation, costings of policy options and the results of economic or statistical modelling. To be as effective as possible, evidence needs to be provided by, and/or be interpreted by, experts in the field working closely with policy makers. This expertise includes economists and statisticians, employed and on a service-wide basis by the Government Economic Service etc, and social researchers, doctors and other scientists employed by departments.’ (ibid.)

to assessing their impact:

‘…effective policy making must be a learning process which involves finding out from experience what works and what does not and making sure that others can learn from it too. This means that new policies must have evaluation of their effectiveness built into them from the start …’ (ibid.)

Analysis and use of evidence is one of the nine professional skills areas in the portfolio of programmes offered by the National School of Government.

At this point it may be worth remarking that policies can risk failure for a whole variety of reasons other than ignorance, rejection or lack of evidence; for example, poor policy formulation, inappropriate balance between incentives and sanctions; inadequate implementation strategy, resistance by those charged with local implementation and/or by those affected, cost, conflict with other policy goalsor strategies, unintended consequneces, difficulty of assessing effectiveness….

Yet it seems to be largely the explicit acknowledgement of hte importance of evidence that gives rise to the epithet ‘modern’ in ‘modern policy-making’ – it signals a clear break from the past when policy was:

‘driven by political ideology, conventional wisdom, folklore and wishful thinking… [representing] the triumph of hope over reason, sentiment over demonstrated effectiveness, intuition over evidence.’ (Davies 1999)

Whether or not this is true or just a caricature, there is no doubt that in the early 2000s the English Department for Education and Employment, as it then was, had high hopes and expectations of research: David Blunkett, then Secretary of State for Education and Skills, made a direct appeal to the research community:

‘social science should be at the heart of policy making…. We need social scientists to help to determine what works and why, and what types of policy initiatives are likely to be most effective.’ (Blunkett 2000)

The Department made a substantial investment in research: as well as setting a large budget (£139m in 2005–06, according to White 2007) for research, it created several dedicated research centres which were part of the academy: the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) was founded to conduct and support systematic reviews of research; another Centre has been investigating the wider benefits of learning and a third studying the economics of education. White (op.cit.) is well worth reading for the unique insight she gives into the Department’s schools research strategy, including an insider’s perspective on the ‘evidence-based policy’ reforms and some of the challenges that followed.

The early 2000s were also the era of the Teaching and Learning Programme, an unprecedentedly large-scale, long-term programme of educational research funded by the Economic and Social Research Council (ESRC). This was specifically established to address the alleged lack of research in education capable of informing policy and practice. In 2000 the DfEE set up the National Educational Research Forum (NERF), with a remit to bring together academic researchers, practitioners and policy-makers to develop a strategic approach to the national educational research and development effort. This body ceased to function in 2006-07, and in 2008 a Strategic Forum for Research in Education was set up.

Local government and the voluntary sector have also been active in promoting research as an aid to decision-making. See, for example:

• http://www.laria.gov.uk/
• http://www.rcuk.ac.uk/innovation/partnership/larci/default.htm
• http://www.idea.gov.uk/idk/core/page.do?pageId=1
• http://www.arvac.org.uk/

TASK

 Draft an outline of the research function and strategy for a new government agency responsible for overseeing the well-being, healthy development and education of children in the early years, ‘a service committed to providing safe and stimulating environments in which all children will be able to develop and grow into resilient, capable, confident and self-assured personalities before they start school.’  Your outline should include: 

• the overall aim of the research function
• possible research and evaluation priorities and reasons for prioritising
• likely purposes of research projects and evaluations
• types of designs that might be might required,
• kinds of data that might need to be collected
• analytical techniques that might be applied

3. The diversification of ‘knowledge’

Despite the injection of financial, intellectual and managerial resources into research (and similar kinds of investment on the part of other countries), an OECD/CERI report Evidence in Education: Linking Research and Policy (OECD/CERI 2007) argues that there continues to be a concern at government and trans-government levels about the relationship between policy and research in education, specifically:

• low levels of investment in educational research;
• generally low levels of research capacity, especially in quantitative research;
• weak links between research, policy and innovation.

Whilst this seems merely to rehearse the criticisms made of educational research as a field in the late 1990s (see especially Hargreaves 1996, Hillage et al. 1998, Tooley and Darby 1998) – to which TLRP, NERF etc. were a response – a closer look shows that the real claim in the report is about a systemic problem that needs revisiting because the policy context has changed.

The OECD/CERI report contains analyses and commentaries by both academic researchers and politicians, and Part Four is worth reading for the variety of political/policy perspectives on research from different countries. Generally, policy-makers are much more hungry for knowledge than in the past, not least because the legitimacy of the state to act in certain ways or at all is constantly being called into question. And much more knowledge is available. One of the changes the report points to is the ‘increased access to information via the Internet and other technologies’. This issue is a crucial one, being about rapidly-changing modes of knowledge creation and access (see, for example, Gibbons et al. 1994 for an influential re-conceptualisation of knowledge production). The consequences of knowledge created and disseminated via the internet for both research and policy-making have been raised by Galvin in a series of thought-provoking papers (for example, Galvin 2004, 2008), which raise stark issues for the academic research community and how it might/should respond to the proliferation of instant, unfiltered ‘knowledge’, compared with universities’ independent, peer-reviewed scholarship whose time-lapse between inception and publication may be years.

For a variety of reasons including timescale, academic and even commissioned research constitutes only one source of knowledge for policy-making – and perhaps a minor one at that. As well as on-line access to instant ‘knowledge’, policy-makers can now consult any one of a large number of think-tanks, which have an explicit policy focus/remit; their knowledge usually travels more swiftly and smoothly into a policy context than academic knowledge. Several of the think-tanks feature education as one of their key areas of expertise. These are a few of the better-known institutes:

• IPPR:’The Institute for Public Policy Research is the UK’s leading progressive think tank, producing cutting edge research and innovative policy ideas for a just, democratic and sustainable world’.
• Demos:’Demos is the think tank for ‘everyday democracy’. Our aim is to put this idea into practice by working with organisations in ways that make them more effective and legitimate.
• Social Market Foundation:’The Social Market foundation is a leading UK think tank, developing innovative ideas across a broad range of economic and social policy. It champions policy ideas which marry markets with social justice and takes a pro-market rather than free-market approach’.
• Policy Exchange:’Policy Exchange is an independent think tank whose mission is to develop and promote new policy ideas which foster a free society based on strong communities, personal freedom, limited government, national self-confidence and an enterprise culture. Policy Exchange is committed to an evidence-based approach to policy-development. We work in partnership with academics and other experts and commission major studies involving thorough empirical research of alternative policy outcomes’.
• The National Foundation for Educational Research, whilst not a think-tank, is a specialist research institute outside the HEI sector: ‘…. since 1946 we have been working to equip decision makers, managers and practitioners with the most innovative thinking, practical research and responsive assessment programmes to underpin the drive towards excellence in education and lifelong learning… We provide high quality, evidence-based research for policy makers, managers and practitioners… Our research contributes to Government policy and our expertise is recognised nationally and internationally.’

TASK

Compare the mission statement, list of publications, staff members and any other relevant information from one or more of these organisations’ websites with those of your own department/institution (if applicable); and then use the Policy Hub website to look at the ways policy is meant to be informed by research. How well does each research provider’s public presence fit with general policy-making objectives, and how might this influence the first impressions of people who want to commission research to inform policy-making?

4. ‘Evidence-based policy’: the proliferation of ‘policy’ and why ‘evidence’ remains a challenge

The examples and quotations in the previous section raise an important and interesting question about the nature of educational research/research in education. On the one hand, there is Blunkett’s (2000) now notorious appeal to research to ‘determine what works’, a notion that has acquired the status of a bête-noire in some academic circles, to the extent that it seems to be assumed that all policy-makers start from a linear and wholly instrumental conception of the relationship between research and policy. On the other hand, ‘research-as-evidence’ is only one of a number of contrasting perspectives on what constitutes educational research, which need to be articulated and distinguished from each other – see the note in the Introduction to this resource.

One position, taken by Loewenberg Ball and Forzani (2007), is that education research is, or should be, a field of specialised knowledge about educational (pedagogic) processes: in effect, they are arguing that educational research is, or should become, an academic discipline with clear epistemic boundaries, rather than a general field which draws upon a wide range of academic disciplines.

Because policy-makers (including in the UK) are – or have been until quite recently – concerned with systems and structures rather than pedagogies, a great deal of commissioned research in education has not been educational in that disciplinary sense:

‘…research that is ostensibly “in education” frequently focuses not inside the dynamics of education but on phenomena related to education––racial identity, for example, young children’s conceptions of fairness, or the history of the rise of secondary schools. These topics and others like them are important. Research that focuses on them, however, often does not probe inside the educational process. Until education researchers turn their attention to problems that exist primarily inside education and until they develop systematically a body of specialized knowledge, other scholars who study questions that bear on educational problems will propose solutions.’ (Loewenberg Ball and Forzani 2007)

Previously, the English Ministry of Education, as a government ‘spending’ department, was concerned with efficient distribution of revenue, matching of supply and demand and so forth; it did not commission much research. But as education has become a greater political priority, continuous reform of, and within, the system has become the norm. The broad foci of education policy in most economically-developed countries in recent years have been on system improvement: raising standards of achievement and reducing ‘achievement gaps’ (significant differences between social groups in terms of their educational performance) and on de-centralising aspects of education management whilst strengthening accountability. Policy ‘levers’ over the last twenty years have included centralising the school curriculum, instituting universal high-stakes testing at each stage of schooling and beyond, deregulating the allocation of pupils to schools, centralising the school inspectorate, publishing the results of school/college inspections, and so forth. These were the motifs of the English Education Reform Act of 1988 and of many other pieces of legislation since. Additional political themes since then have included the reform of vocational education and qualifications, the extension of higher education, the establishment of professional standards for practitioners, and so on. Add to this that, in England, education responsibilities have been separated into two departments, one for integrated children’s services (the Department for Children, Schools and Families) and the other for ‘innovation, universities and skills’, and it becomes obvious how far the education policy agenda has sprawled. Their current research programmes included (chosen at random on 22 January 2009): DCSF evaluations of the supported housing pilot for teenagers, investment in initiatives designed to improve teacher workforce skills in relation to SEN and disabilities, parenting early intervention programme, disabled children in IB budgets, intensive teens and toddlers pilot project; DIUS projects on STEM graduates in non-STEM jobs, understanding the needs of parents regarding relationship support, evaluation of the UK-Russia Bridge Higher Education Partnership Programme, all at: http://www.dcsf.gov.uk/research/programmeofresearch/index.cfm?type=0 The sheer range and diversity of research and evaluation projects being commissioned to inform and evaluate policy is impressive (or slightly mind-boggling, depending on your point of view).

(It’s interesting that only recently have policy-makers in England turned their attention to policy-led pedagogical approaches, in the form of the national strategies in primary and secondary schools – ‘one of the most ambitious change management-programmes in education’ according to Teachernet, the government website for teachers. It can be argued that this completes central government appropriation of the domains of curriculum, assessment and pedagogy that were once thought to be the responsibility and prerogative of the teaching profession.)

So it is not only the signature requirement of ‘modern policy-making’ for evidence but also the proliferation of government-led activity that has resulted in an increased effort around evidence-gathering and ‘knowledge management’ – which has been required to assist in the design, management and evaluation of policies and initiatives, on the part not only of central government but also of the agencies and non-departmental government bodies (NDPBs) which have been set up one after the other as the implementation and management of policies and initiatives has become increasingly complex.

And perhaps this all helps to explain the phenomenon noted in the previous section: the increasingly wide range of organisations – university departments of education, think-tanks, specialist centres, private sector companies, consultants – that have been commissioned by national and local government and NDPBs to provide evidence and research in education since 1988. An early example of this trend was a report for the Department of Education and Science by PricewaterhouseCoopers on the local management of schools, which shocked educationists at the time but would now attract little attention. Indeed, if – in any given case – the prime need is for knowledge about gender inequality, organisational leadership, nutritional needs of adolescents; or for skills in sample surveys, ethnography, democratic consultations, it is at least understandable if those commissioning research decide to turn to specialists in those fields/methods.

The changed context possibly also helps to account for the capacious typology of ‘evidence’ listed in ‘Professional Policy Making for the Twenty First Century’ (and already quoted above), viz.:

• expert knowledge
• existing domestic and international research
• existing statistics
• stakeholder consultation
• evaluation of previous policies
• new research, if appropriate
• secondary resources, including the internet;
• analysis of the outcome of the consultation;
• costings of policy options
• results of economic or statistical modelling

If and when educational researchers think of themselves as providing research for policy, they probably think initially in terms of only one or two of these, expert knowledge and new research. The list (which is actually a mixture of method and category) also contains reference to two kinds of evidence which have become emblematic of the debate about evidence-based education amongst educational researchers: (reviews of) existing research and ‘statistics’. (See Andrews 2007 for a helpful and practical discussion of what counts as evidence.)

Although it is not clear from the above list whether ‘statistics’ refers to the analysis of administrative numerical datasets or to quantitative research methods or both, policy-makers pretty constantly express a desire for more/better quantitative skills amongst educational researchers (and this is reiterated in the OECD/CERI report). The educational research community in the HEI sector has responded in two ways, by attempting to build capacity in quantitative research skills through workshops, training programmes, materials, specialised centres (see, for example, the National Centre for Research Methods ); and also through debate on the relevance of quantitative approaches to research in education (with some academics questioning the whole quantitative paradigm on the grounds that it is ‘positivist’ and can therefore lead only to reductionist conclusions, others asserting that a judicious deployment of different quantitative approaches can reliably generate ‘secure findings’, and yet others arguing that the quantitative/qualitative distinction describes ideological rather than methodological alternatives).

Turning to reviews of existing research, one of the strands in the evidence-based education argument is the need to take stock of, and build on, what we know already – a seemingly common-sense position (see, for example, Davies 2004) that has led to the development of a complex apparatus for ‘systematic’ reviews (see the EPPI-Centre) that very much follows an existing medical model (the Cochrane Collaboration). Taken together with the issue of quantitative approaches, the development of the ‘systematic review’ methodology has provided further grounds for researchers to question whether there is any merit in proposing that educational is, or should aspire to be, ‘scientific’ in the sense put forward by, for example, the Committee on Scientific Principles for Education Research (National Research Council 2002).

Policy-makers’ stance is perhaps more easily understood in the context of their justifiable preoccupation with scrutinising what solutions have been tried with what effect at other times and elsewhere, with monitoring and evaluating systemic change over time, and with getting some grip on the horrible complexity of cause-and-effect – though it would be quite wrong to suggest that commissioned research, even at government level, consists mainly of systematic reviews, random controlled trials and statistical analysis. The process of evidence-gathering is more pragmatic than ideological; much commissioned research is mixed method or wholly qualitative, and often has a conceptual (if not theoretical) dimension, as a quick glance at the published research reports on the Department website, for example, will confirm.

From the research community’s point of view, one could argue that the evolving debate about evidence-based education, inextricably linked as it is with discussions about values, paradigms and methodologies, has raised the game as well as the temperature of educational researchers. As Pollard noted (2007), a classic question has been whether social scientists should ‘take’ or ‘make’ the problems they study. Should they respond to the agendas and priorities of others (such as funders or governments), or position themselves to develop more independent analyses of social issues? What should be the role of the university in a free society, the intellectual in a democracy, or the social scientist in his or her community?

Some researchers have espoused the evidence agenda, albeit with some qualifications (mostly to do with policy-makers’ apparent disregard of the evidence in certain cases) or sometimes perhaps more in practice (by seeking research commissions from government) than in theory. Some have commented on the need to inject some intellectual hygiene into the discourse, for example by clarifying what is meant by ‘evidence’ or ‘scientific’. Others believe that research should in principle maintain a critical stance on, and a healthy distance from, policy-making, in order to safeguard academic independence; still others question the validity of an ‘evidential basis for policy’ as an idea or enterprise per se. Rather than try, and fail, to do justice in this resource to the full range of stimulating and learned books and papers that have been produced over the last decade or so, we think it is more useful to suggest key reading on the main issues, which will themselves offer plenty of further sources. Of those listed in the bibliography below, you are particularly recommended to read: Ball 1995, Oakley 2000 (long but gripping), Davies 2000, Oxford Review of Education 2000, Hammersley 2002, Slavin 2002, National Research Council 2002, Pawson 2002 and 2006 (well worth reading, since his critique is also a serious attempt to engage strategically with the problems of policy design and evaluation), Nutley 2003, Thomas and Pring 2004, Andrews 2007, Whitty 2007, Bridges et al. 2008a, Biesta 2009. See also Martyn Hammersley’s useful booklist.

One of the obvious conclusions to draw from this brief overview is that ‘research’ means different things to different people. Generalising wildly, one might say that for policy-makers research does need to discover and disseminate ‘what works and why [the second half of Blunkett’s sentence is often omitted in citations]’ to raise standards and improve the ‘effectiveness’ of schools – and, in the process, to help institutions become ‘learning organisations’, capable of ‘effective knowledge management and transfer’; research is thus an organisational tool for becoming ‘data/information-rich’, exploring and comparing ‘new solutions’, competing for influence… The inverted commas signify that it is precisely these terms which most educational researchers would wish to question and problematise from a variety of theoretical and ethical perspectives. Moreover, as Bridges et al. (2008b) and Biesta (2009) argue, research in education has multiple affordances: it is not always oriented towards (let alone can provide) solutions to educational problems, but may – sometimes more usefully – put the problem in a broader historical or social context. Thus it may confer deeper understanding by correcting assumptions, providing insights into people’s experiences, revealing the complex nature of the problem, and keeping the question of educational values explicit – education being a value-laden domain in which answers to policy questions can never be purely technical.

TASK

Based on your reading of the key texts, summarise the arguments that have been used to promote the evidence-based education agenda, and those that take a critical view of the same issues. Which do you think are the most cogent points to emerge on both sides?

5. Research and policy: marriage, divorce or just good friends?

‘What counts is what works‘ but has anything changed in the relationship betwen research and policy since 2000? Here is another extract from ‘Professional Policy Making for the Twenty-First Century’, published in 2000:

‘Whilst there is plenty of research available in areas such as education, social services and criminal justice, the coverage is patchy and there is little consensus amongst the research community about the appropriateness of particular methodologies or how research evidence should be used to inform policy and practice. These factors perhaps contribute to our finding that, although there are examples of good practice, in some areas of policy the generation and use of information and research in policy making is not as strong as it needs to be to support the Government’s pragmatic approach…. The two key issues are:

• the need to improve departments’ capacity to make best use of evidence;
• the need to improve the accessibility of the evidence available to policy makers’.

Representatives bodies of the academic community, like the Academy of Social Sciences and the British Academy, have undoubtedly wanted to accept the challenge of providing relevant and high quality research and to make it accessible for policy purposes, even if they wanted to urge caution about how this could be done (and even if some researchers have had misgivings about the whole project). Here is an extract from the report from the Commission on the Social Sciences, three years later:

‘There are significant problems with the exploitation of social sciences research in government, local government, commerce, the voluntary sector and the media. These come about because of ‘interface management’ and communication problems, though the caution of some academics towards close engagement with practitioners is a source of great disappointment to many users of social science research.’ (Academy of Social Sciences 2003)

The picture had not improved by 2007: here is another excerpt from the OECD/CERI report cited earlier:

‘…available information often does not provide the elements necessary for decision-making, either because the rigorous research relevant to policy needs has not been conducted, or the research that is available does not suggest a single course of action.’

And, in 2008, was anyone still listening?

‘It is vitally important that UK policy makers are able to make use of all that humanities and social science research has to offer. By any measure, UK research in the humanities and social sciences is first-class. It generates evidence and findings of high salience for policy makers. This report illustrates many of the central challenges facing policy makers in using that research. It also gives examples of the various ways in which humanities and social science researchers can help policy makers to respond to these challenges.

Our findings reveal serious concerns that policy makers are not realising the full potential of the contributions that humanities and social science research can make to public policy making. Policy makers and academic researchers alike are agreed that more should be done to strengthen that contribution. This report illustrates these problems and contains proposals for consideration by Government and other bodies.’ (Extract from President’s foreword by Baroness Onora O’Neill, British Academy 2008.)

But perhaps the whole project is misconceived?

‘There is a widespread assumption that research provides an “evidential basis” for policy or, more acceptably perhaps, that research “informs” policy. The notion of research providing a basis for policy is especially problematic in so far as it suggests that the process begins with research which then points to the required policy. This is an empirically and logically unsound view of the nature of policy and its construction. Policy is an ongoing process: it is not a vacuum waiting to be filled. It has a history and a contemporary social political context. It is there before the research comes along: it is not waiting for research to bring it into existence… Research may arouse interest, provoke debate, confirm prejudice, give new insight, challenge pre-existing beliefs but it will… rarely even be the predominant informing resource [for policy]…’ (Bridges et al. 2008b)

Or is it rather more easily understood in terms of:

‘… the familiar collision between the culture of free inquiry that bristles at governmental encroachment and the equally compelling culture of democratic accountability that demands evidence that public monies are wisely spent’ (Feuer et al. 2002)?

Before making up our minds, however, perhaps we should make good the biggest lack in our knowledge:

‘the relationship between policy research, the evidence it generates and the policy process remains complex and not well understood… in these debates, less attention has been paid to how we conceptualise the demand side of the equation – the nature of policy-making’ (Burton 2006).

TASK

Read the reports from the Academy of Social Sciences, OECD/CERI and the British Academy and then summarise the main issues they identify as if from (i) a policy perspective (ii) an academic perspective, noting any differences between the findings from each of these bodies.

Now look particularly at the recommendations they make and compare these with some of the objections raised by academic researchers to the evidence-based education agenda. In your opinion, do they take sufficient account ‘both of the devolved and iterative ways in which policy-making is now being done… and of the more engaged, speculative and scenario-building role for research…’ (Saunders 2007)? Can/should anything more be done now to improve the links between research and policy? If so, briefly summarise your suggestions.

6. Knowledge, skills, values and attitudes: how researcher identity is constructed in policy environments

Saunders (2007) opens her chapter ‘Go-betweens, gofers or mediators? exploring the role and responsibilities of research managers in policy organisations’ with this observation:

‘National policy agencies in education (including but not limited to the DfES) employ between them large numbers of qualified researchers (of whom the author is one), whose roles include designing, commissioning, managing, undertaking, collating and mediating research evidence in and for a policy environment. The interventions of such people are, or ought to be, crucial to the ways in which research enters (or is detached from) the discourses of policy-making. However, not much in the literature on policy-makers’ use of research refers to the role of these professionals specifically and directly, and what there is seems to rest on assumptions and assertions rather than empirical-analytical description.’

She goes on:

‘… I have so far been able to find very little in the literature on the relationship between research and policy that refers specifically and directly to the very practical matter of who these people are, what they do, and what kinds of professional expertise, values and relationships they draw on. What literature there is seems to rest on assumptions and assertions rather than empirical-analytical description and/or tends to bracket all kinds of policy staff and research advisers together and seems to assume that there is no stronger impulse guiding their work than political naïveté or expediency (see, for example, Humes and Bryce, 2001; Temple 2003).’

To date, there does not appear to be any more recent published literature on this area. So another way of attempting to understand the knowledge, skills, values and attitudes expected of this group of professional staff is to examine the frameworks, programmes and competences for their training and development (which have already been mentioned in Section 1 above).

TASK 1

The key sites to look at are:

• http://www.gsr.gov.uk/

‘The Head of the Government Social Research service and supporting office (GSRU) provides strategic leadership to the Government Social Research service and supports it in delivering an effective service.’  Look also at the GSRU strategic plan and organization chart.

• http://www.gsr.gov.uk/downloads/professional_development/cpd/cpd_handbook.pdf

GSR competencies and professional skills, for example:‘Critical analysis and decision making:Critically evaluates data and information with accuracy and perception, and is able to synthesise and use data from a variety of different methods appropriately. Makes sound, evidence based decisions (and/or helps others do so); assesses risk and defends decisions and action; responds effectively to unforeseen situations.’

• http://www.nationalschool.gov.uk/policyhub/magenta_book/ [2003 updated 2007]

Sections include:

• What do we already know? Harnessing existing research
• What do the statistics tell me? Statistical concepts, inference and analysis
• How are the data collected? Data collection and survey design
• Why do social experiments? Experiments and quasi-experiments for evaluating government policies and programmes
• How do you know why (and how) something works? Qualitative methods of evaluation

• http://www.nationalschool.gov.uk/individual_development/professions.asp

‘The Government Social Research Unit runs a number of professional development courses in Policy Analysis and Evaluation.  These courses have a high academic content and are specifically designed for researchers and other analysts working within government, and policy makers who work closely with researchers. They are designed to provide students with an understanding of the major quantitative and qualitative research skills relevant to designing, analysing and evaluating government policy.’

First of all, examine these frameworks, guidelines and competences carefully, and consider whether and how they lead to particular conceptions of research and of researcher identity – in terms of required/desired knowledge, skills, values and attitudes – within a policy environment.

Secondly, based on a combination of reading, talking to others and your own experience, consider – in the context of someone taking a secondment into the DCSF from university, say – the main similarities and differences between the ‘constructed identities’ of academic researchers in university departments of education and social researchers in government, especially in relation to the creation and critiquing of knowledge. You may find it helpful to consult Brown 2007 in formulating your ideas.

TASK 2

Saunders identifies these as some of the questions that need to be explored about researchers working in policy-making organisations:

• ‘an exploration of which kind(s) of “conceptual framework” – policy studies, sociology, politics of education, etc. – might be most able to provide a sensitive and plausible way of thinking about interactions between policy and research, and about those who manage or broker them; in particular we need to understand how to conserve an appreciation of the “good faith” of those who work in these roles… whilst also allowing for the ways in which their self-accounts are shaped by the boundaries and requirements of those roles;
• the exploration of whether research can and should be conceived of (and flourish in practice) as an independent professional discipline within policy organizations or whether it is more usefully and pragmatically to be thought of and organized as an internal service, possibly with the full apparatus of service level agreements.’

Are these the most pressing questions? What others would you add, having looked at the government’s training and development material for researchers? What research methodologies would be most suitable/feasible to explore them?

Further reading and bibliography

ACADEMY OF SOCIAL SCIENCES. (2003). Great Expectations:  the Social Sciences in Britain. Downloadable at: http://www.acss.org.uk/docs/GtExpectations.pdf 

ANDREWS, R. (2007). ‘What counts as evidence in education?’ Online at Teacher Training Resource Bank. Downloadable from: http://www.ttrb.ac.uk/viewArticle2.aspx?contentId=14608 or as a PDF at: http://www.ttrb.ac.uk/attachments/38ba1867-f5d9-4c5f-943c-3218a7399724.pdf

BALL, S. J. (1990).  Politics and Policy Making in Education.  London:  Routledge.

BALL, S. J. (1995) ‘Intellectuals or technicians? The urgent role of theory in educational studies’, British Journal of Educational Studies, 43, 3, 255-71.

BERA. (2003).  Educational Policy and Research across the UK.  Report of a BERA Colloquium held at the University of Edinburgh, 7–8 November 2002.  Nottingham:  British Educational Research Association.

BERA. (2008). ‘Philip Cowley: your man in the Treasury’, Profile, Research Intelligence 105, November.

BLUNKETT, D. (2000). ‘Influence or irrelevance: can social science improve government?’, Research Intelligence 71.

BIESTA, G. (2007). Why ‘what works’ won’t work. Evidence-based practice and the democratic deficit of educational research. Educational Theory. 57, 1, 1–22 .

BIESTA, G. (2009). ‘Educational research, democracy and TLRP.’ Keynote presentation at the TLRP Research Conference, London, Institute of Education, 19 March. Downloadable at: http://www.tlrp.org/conference/2009/downloads.html

BRIDGES, D., SMEYERS, P. and SMITH, R. (Eds) (2008a). ‘“Evidence-based Educational Policy”: What Evidence? What Basis? Whose Policy?’. Journal of Philosophy of Education, 42, Suppl. 1, whole issue.

BRIDGES, D., SMEYERS, P. and SMITH, R. (Eds) (2008b). ‘Educational research and the practical judgement of policy makers’, Journal of Philosophy of Education, 42, Suppl. 1., 5– 14.

BRIDGES, D., CONROY, J., DAVIS, R., ELLIOTT, J., ENSLIN, P., GRIFFITHS, M., MCLEOD, G., FOREMAN PECK, L., LUKES, D., MURRAY, .J, OANCEA, A., PRING, R., SAUNDERS, L., SMEYERS, P., SMITH, R. and WATTS, M. (2008c). ‘Educational research and policy: epistemological perspectives.’ London: TLRP. Online at http://www.tlrp.org/capacity/rm/wt/pesgb1.html

BRITISH ACADEMY. (2008). Punching Our Weight: The Humanities and Social Sciences in Public Policy Making. Published online 17 September 2008: http://www.britac.ac.uk/reports/wilson/index.cfm

BROWN, A. (2007). Understanding researchers’ career development. London: TLRP. Online at http://www.tlrp.org/capacity/rm/wt/brown/brown2.html

BURTON, P. (2006). ‘Modernising the policy process: making policy research more significant?’, Policy Studies, 27, 3, 173–95.

CHITTY, C. (2004). Education Policy in Britain. Basingstoke: Palgrave Macmillan.

DAVIES, P. (1999). ‘What is evidence-based education?’, British Journal of Educational Studies, 47, 2, 108–21.

DAVIES, P. (2000). ‘The relevance of systematic reviews to educational policy and practice’, Oxford Review of Education, 26, 3-4, 365-78.

DAVIES, P. (2004). ‘Is evidence-based government possible?’ The 2004 Jerry Lee Lecture, Campbell Collaboration Colloquium, Washington DC, 19 February.

DAVIES, H.T.O, NUTLEY, S.M. and SMITH, P.C. (2000). (Eds) What Works? Evidence-based Policy and Practice in Public Services.  Bristol:  The Policy Press.

ERICKSON, F. (2005). ‘Arts, humanities and sciences in educational research and social engineering in federal education policy’, Teachers College Record, 107, 1, 4–9.

FEUER, M.J., TOWNE, L. and SHAVELSON, R.J. (2002).  ‘Scientific culture and educational research’, Educational Researcher, 31, 8, 4–14.

FLYVBJERG, B. (2001).  Making Social Science Matter.  Why Social Inquiry Fails And How It Can Succeed Again.  Cambridge:  Cambridge University Press.

FURLONG, J. and OANCEA, A. (2005). Assessing Quality in Applied and Practice-based Educational Research: a Framework for Discussion (ESRC Report RES-618-25-6001). Oxford: Oxford University Department of Educational Studies.

GALVIN, C. (2004). ‘The making of public policy in a digital age: some reflections on changing practice’.  Paper presented at Market Research Society Conference ‘Social Policy Research in the Information Age’, London, 17 February.

GALVIN, C. (2008). ‘“But all the wrong people are here…”:  the emerging policymaking mode and its challenge for academic policy research: a commentary’,
Cambridge Journal of Education, Special Issue: Knowledge Transformation and Impact, 38, 1,  121–30.

GIBBONS, M., LIMOGES, C., NOWOTNY, H., SCHWARTZMAN, S., SCOTT, P. and TROW, M.  (1994).  The New Production of Knowledge.  London:  Sage.

HAGGER, H. and FURLONG, J. (2006). ‘”Taking the evidence”: comparing a national policy review with policy research: the Review of Initial Teacher Training Provision in Wales’, presentation at Oxford Department of Education Studies, 30 October.

HAMMERSLEY, M. (2002). Educational Research, Policymaking and Practice, Paul Chapman.

HAMMERSLEY, M. (2004). ‘Some questions about evidence-based practice in education’, in THOMAS, G. and PRING, R. (Eds.) Evidence-Based Practice in Education. Maidenhead: Open University Press.

HARGREAVES, D. (1996). Teaching as a Research-based Profession:  Possibilities and Prospects.  (Teacher Training Agency Annual Lecture 1996).  London: TTA.

HILLAGE, J., PEARSON, R., ANDERSON, A. and TAMKIN, P. (1998). Excellence in Research on Schools.  London: DfEE.

HUMES, W. and BRYCE, T. (2001). Scholarship, research and the evidential basis of policy development in education’, British Journal of Educational Studies, 49, 3, 329–52.

LEVIN, B. (2001). ‘Knowledge and action in education policy and politics’, presentation at conference on empirical issues in Canadian education, Ottawa, November.  Downloaded 24 July 2003 from: http://home.cc.umanitoba.ca/~levin/sweetmanconf_files/frame.htm

LEVIN, B. (2003). ‘Research, policy and the state in education’, presentation to Deakin University, 28 March.  Downloaded 24 July 2003 from: http://home.cc.umanitoba.ca/~levin/deakin%20mar03_files/frame.htm

LEVIN, B. (2004). ‘Making research matter more’, Education Policy Analysis Archives, 12, 56, 1–17.  Downloadable at: http://epaa.asu.edu/epaa/v12n56/v12n56.pdf

LOEWENBERG BALL, D. and FORZANI, F. M. (2007). ‘What makes education research “educational”?’, Educational Researcher, 36, 9, 529–40

MOSS, G. and HUXFORD, L. (2007). ‘Exploring literacy policy-making from the inside out’.  In: SAUNDERS, L. (Ed) Educational Research and Policy-making:  Exploring the Border Country Between Research and Policy.  London:  Routledge.

MUNN, P. (2005). ‘Researching policy and policy research’, Scottish Educational Review, 37, 1, 17–28.

NATIONAL AUDIT OFFICE. (2003). Getting the Evidence: Using Research in Policy Making. London: National Audit Office. http://www.nao.org.uk/publications/0203/using_research_in_policy_makin.aspx

NATIONAL EDUCATIONAL RESEARCH FORUM. (2000). Research and Development in Education: a National Strategy Consultation Paper, Nottingham, National Educational Research Forum.

NATIONAL RESEARCH COUNCIL. (2002).  Scientific Research in Education.  Committee on Scientific Principles for Education Research.  Washington, DC:  National Academy Press.

NISBET, J.D. and BROADFOOT, P. (1980).  The Impact of Research on Policy and Practice in Education. Aberdeen: Aberdeen University Press.

NUTLEY, S. (2003).  ‘Bridging the policy/research divide:  reflections and lessons from the UK.’  Keynote paper presented at ‘Facing the future:  engaging stakeholders and citizens in developing public policy’, National Institute of Governance Conference, Canberra, 23 April.

NUTLEY, S.M. and DAVIES, H.T.O. (2000).  ‘Making a reality of evidence-based practice’. In:  DAVIES, H.T.O., NUTLEY, S.M. and SMITH, P.C.  (Eds)  What works?  Evidence-based policy and practice in public services.  ristol:  The Policy Press.

 NUTLEY, S.M. and WEBB, J. (2000).  ‘Evidence and the policy process’. In:  DAVIES, H.T.O., NUTLEY, S.M. and SMITH, P.C.  (Eds)  What works?  Evidence-based policy and practice in public services.  Bristol:  The Policy Press.

OAKLEY, A. (2000). Experiments in Knowing: Gender and Method in the Social Sciences.  Cambridge: Polity Press.

ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT, CENTRE FOR EDUCATIONAL RESEARCH AND INNOVATION. (2007). Evidence in Education:  Linking Research and Policy. Downloadable from:

http://www.oecd.org/document/56/0,3343,en_2649_35845581_38796344_1_1_1_1,00.html

OXFORD REVIEW OF EDUCATION (2000), The Relevance of Educational Research, 26, 3-4 (whole issue).

PAWSON, R. (2002). ‘Evidence-based policy: the promise of “realist synthesis”’, Evaluation, 8, 3, 340-58

PAWSON, R. (2006). Evidence-Based Policy: A Realist Perspective.  London:  Sage Publications. 

POLLARD, A. (2007). ‘Foreword’. In: SAUNDERS, L. (Ed) Educational Research and Policy-making:  Exploring the Border Country Between Research and Policy.  London:  Routledge.

PRING, R. (1995). ‘Educating persons: putting “education” back into educational research (the 1995 SERA Lecture), Scottish Educational Review, 27, 2, 101-12

SAUNDERS, L. (2003).  ‘Becoming an evidence-based professional organisation:  en route from research projects to organisational knowledge’.  Paper presented at the Market Research Society conference, Challenges and Opportunities in Public Policy Research, London, 12 February.

SAUNDERS, L. (2004).  Grounding the Democratic Imagination:  Developing the Relationship between Research and Policy in Education.  Professorial Lecture.  London: Institute of Education.

SAUNDERS, L. (2005a).  ‘Research into policy doesn’t go? A response to “Interesting times, and interested research”, British Educational Research Journal Editorial, 30, 4, 475–476’,  British Educational Research Journal, 31, 1, 2-4.

SAUNDERS, L. (2005b).  ‘Research and policy: reflections on their relationship’, Evidence and Policy: a Journal of Research, Debate and Practice, 1, 2, 383–90.

SAUNDERS, L. (2007). ‘Go-betweens, gofers or mediators? Exploring the role and responsibilities of research managers in policy organisations’. In: SAUNDERS, L. (Ed) Educational Research and Policy-making:  Exploring the Border Country Between Research and Policy.  London:  Routledge.

SAUNDERS, L. (2008) ‘Educational research commissioned by/for policy audiences.’ London: TLRP. Online at http://www.tlrp.org/capacity/rm/wt/saunders1.html

SLAVIN, R. E. (2002). ‘Evidence-based education policies: transforming educational practice and research’, Educational Researcher, 31, 7, 15-21.

SOCIAL RESEARCH ASSOCIATION. (2002).  Commissioning Social Research:  A Good Practice Guide.  London:  SRA.  Downloadable at http://www.the-sra.org.uk/commissioning.pdf

SOMEKH, B. (2007). ‘The interplay between policy and research in relation to ICT in education in the UK:  issues from twenty years of programme evaluation’.  In: SAUNDERS, L. (Ed) Educational Research and Policy-making:  Exploring the Border Country Between Research and Policy.  London:  Routledge.

St. CLAIR, R. (Ed) (2009 forthcoming).  Education Science: Strategies and Challenges in the Production of Knowledge. Rotterdam: Sense.

TEMPLE, P. (2003).  ‘Educational research and policymaking: findings from some transitional countries’, London Review of Education, 1, 3, 217–28.

THOMAS, G. and PRING, R. (Eds) (2004). Evidence-Based Practice in Education.  Conducting Educational Research series.  Maidenhead: Open University Press.

TOOLEY, J. and DARBY, D. (1998). Educational Research: A Review. London: OFSTED.

WEISS, C.H. (1991). ‘Policy research: data, ideas or arguments?’  In:  WAGNER, P., WEISS, C.H., WITTROCK, B. and WOLLMAN, H. (Eds)  Social Sciences and Modern States:  National Experiences and Theoretical Crossroads.  Cambridge:  Cambridge University Press.

WHITE, V. (2007). ‘Schools research in the English Ministry of Education: an inside view.’ In: SAUNDERS, L. (Ed) Educational Research and Policy-making:  Exploring the Border Country Between Research and Policy.  London:  Routledge.

WHITTY, G. (2002).  Making Sense of Education Policy.  London: Paul Chapman.

WHITTY, G. (2007). ‘Education(al) research and education policy-making: is conflict inevitable?’ In: SAUNDERS, L. (Ed) Educational Research and Policy-making:  Exploring the Border Country Between Research and Policy.  London:  Routledge.

WILLINSKY, J. (2000).  If Only We Knew:  Increasing the Public Value of Social-Science Research.  London:  Routledge.

Return to top of page