An Interview with Dr Chris Brown: Making Evidence Matter
1) Dr. Brown, you have just published a book on “ Making evidence matter: new perspectives on evidence-informed policy making in education” (London, IOE Press). What brought this about ?
Well I’ve been interested in the notion of being evidence-informed for some time now, specifically within the field of educational policy-making. Before joining the Institute of Education (University of London) as a research fellow, I was a researcher but one working inside of government (I was Head of Research for the Training and Development Agency for Schools). What interested most in this role was why, how and when research findings were taken on board as part of the policy decisions my colleagues were making.
Often failure to use evidence is placed firmly at the door of researchers and their ‘inabilities’ to communicate effectively. As a result academics, in addition to their day to day business of writing journal articles, have been encouraged to ensure that their research outputs are both digestible and applicable: that what they write can not only be easily understood, but that it is also immediately ‘policy ready’. I’d argue however that efforts to do so often result in unhappy frustration. This is because whilst useful, these two qualities alone are unlikely to lead to a greater uptake of research by policy-makers: ideas may still sit outside of what is (at any given point in time) politically acceptable to policy-makers or policy-makers may simply fail to see any need to act on what is presented.
I wanted to see if there was a way to improve this situation and so the subject formed the basis of my doctorate (which I undertook between 2009-2011). Since then I have continued to expand on my analysis and understanding, in particular by examining notions of expertise. The result is Making Evidence Matter, which considers how and the means through which being evidence informed can become a more regular way of life within the policy-development process.
2) When you say “evidence“- are you talking about public opinion, or about what you read in the Daily Mail or data collected on line, or some other venue ?
A favourite quote of mine was made by the ex Liberal Democrat MP, Evan Harris, who introduced evidence into a parliamentary debate on cancer with the following statement: “The honourable member for Braintree cited evidence from The Sun, so I want to refer to a recent edition of the British Medical Journal. Such extremes aside, policy-makers will consider a range of evidence types. In thinking about evidence-informed policy-making, I have had to think long and hard about what evidence means, and importantly, what ‘best’ evidence is.
In terms of the former, I typically like to think of evidence as being data that has been gathered through the research process, which has been interpreted and which subsequently can be used to address a particular policy issue. In other words it is not anecdote, it is something that has been gathered in a systematic and rigorous way. In addition, by using the term ‘interpreted’, I specifically mean that which is presented is not simply the raw data produced by the research process, but also the significance ascribed to the data by the researcher (which could range in scope from simple conclusions to actionable recommendations).
There is a danger of course that researchers will tend to interpret evidence according to the biases inherent within them; correspondingly that where studies do not provide the most effective depiction of the social world, of policy-makers following the findings of individual studies simply because they ring true. This means we also have to understand what ‘best’ evidence might mean. To me, the notion of best evidence is about aggregation: if a tranche of researchers of differing perspectives explore a common area or topic and do so via a number of different methodologies and approaches and if consensus and a given course of action emerges, this has to be better evidence that anything that emerges from a single study.
3) Sometimes, the “evidence“ is counter –intuitive—“everyone knows“ that alcoholism is a problem- but the data the evidence of people seeking treatment is minimal. What is a social scientist to do?
I think there are two approaches here: the first is to try and work in partnership with policy-makers to try and help them develop policy solutions that tackle some of the key issues of the day (one of the key messages of Making Evidence Matter). The second, I suppose, concerns how social scientists might promote research messages more generally. Here, what is required is substantial ground-work to enhance the ‘social robustness’ of any idea – to promote its importance and the need to act as a result. Efforts to enhance social robustness can be directed via the general media, social media or through cultivating links with special advisors and others who matter, but the ultimate endgame of this action is to advance ideas towards what Malcolm Gladwell describes as the ‘tipping point’: ensuring issues enter and dominate the mainstream and so must be addressed.
As well as relating to general ideas, however, we can also direct similar efforts towards promoting ourselves as experts, whose advice should be sought. Again, the result is the same, with those considered to be worth listening to finding it easier to catch the ears of policy makers than those who are not (as can be evidenced by those particularly skilled in this approach
4) Often “ evidence “ is surprising- for example, certain problems seem to be located in certain specific parts of a country. The data indicates that the “ pervasive “ problem is really located in the north, northwest part of a country. What does this data tell policy makers?
To me it highlights the importance of context and the tension between policy-makers wanting to produce solutions that can be rolled out across the country and the need for areas, or even individual schools, to be able to marry a given imperative to the issues they face. Vital too is the capacity and/or support structures that individual areas might have available to them to meet such an imperative – this will affect the ability of policies to deliver their intended results.
5) Now, I have visited your Houses of Parliament and Houses of Lords- how do you get the data, the evidence to the right people there?
One of the people I interviewed for my doctorate discussed the notion of echo chambers (the congruence of like minded people who establish ways of thinking about social issues) and this opened up a whole vein of analysis for me: how we might come to understand the powerful role lobby groups and think tanks have in informing policy. I later experienced the power of the echo chambers when I worked in England’s Ministry of Justice.
For example, before the 2010 the notion of ‘restorative justice’ was roundly rubbished by New Labour. Following the election restorative justice was made in the (2010) manifesto of the Conservative/Liberal Democrat coalition government: The Coalition: our programme for government’(published on 20th May 2010).
This change in policy-direction was the direct be the result of the lobbying efforts by England’s Restorative Justice Council and the Association of Chief Police Officers as to its efficacy of restorative justice along with exposure to evidence to reinforce this message.
The press of course can also be key: as I mention earlier getting a message to a ‘tipping point’ means that it is more likely to be noticed and acted upon.
6) Let’s talk a little about educational research. The general populace seems to feel that schools and teachers are underperforming. What does the data say?
Pass! If the government wants the data to back up its version of events it will be able to find it somewhere.
7) When you look at “ data and evidence” can you validly and reliably compare the educational results of 1980 to 1990 to 2000 and 2010?
It seems unlikely: political machinations seem to regularly raise the bar in terms of what good performance is, not only this but in the UK we have changed our curricula numerous times and also our examination system from norm referenced marking to criterion referenced marking. Combined these things reduce the ability to meaningfully compare. Realistically, however, politicians are not likely to seek such a comparison: it often makes a great sound-bite to paint a rosy picture of the past, a catastrophic vision of the future and then to show how current policies have diverted us away from this.
8) How can researchers better influence education policy? Or even should they? Should these researchers who have never been in a school and never taught a day in their lives really tell others how to impact policy?
First, we need to think about who you classify as a policy maker! With Making Evidence Matter I have looked at those in central government – loosely referred to in England as those in Westminster. Here there will be two types of policy-maker: politicians, those elected to carry out a political mandate and civil servants the (apolitical) bureaucrats who create the policy text. It is rare that either the Secretary of State for Education of the officials in their department will have a previous experience in terms of running a school or in relation to hands-on teaching, yet these people are responsible at a macro level for many of the things that impact on schools.
Second and perhaps most importantly, the underpinning message of the book is about partnership: specifically that being evidence-informed is not about researchers passing on chunks of knowledge to teachers, policy-makers or others. Instead I’m putting forward the argument that we [researchers] can bring what we know to the table, but this will only have meaningful impact if we can find ways for policy-makers or teachers to reflect on it and to combine it with what they know (i.e. with practical, tacit knowledge) to produce context relevant and culturally appropriate solutions. This is generally termed ‘knowledge creation’ and it has benefits for researchers too because they can see how generalized ‘best practice’ might play out in actuality.
My proposed way forward therefore is the establishment of policy-learning communities and the use of appropriate tools and processes within them to ensure that knowledge creation actually happens (i.e. we know a lot about what affects the effective running of these communities within schools and this learning needs to be transposed to the world of policy). Membership of these communities would comprise not only researchers and policy-makers but also others whose perspectives will be vital if policy is to be enacted properly. Clearly then this involves teachers but there will be other ‘street level bureaucrats’ with views that need to be considered. Also vital is ensuring that support and expectation is there so that policy-makers treat participation in these communities as an integral part of their role.
9) What have I neglected to ask?
A key question I think is how you resolve the tension between politics and being evidence-informed? My response is that neither being totally evidence-based or totally ideologically led is desirable. The former places optimality above values and what might be desirable, the latter ignores evidence that might improve how a given solution plays out when implemented. In a sense then, when we talk about being evidence informed, we have to remember that there is a boundary of acceptable knowledge that policy-makers might consider. Clearly evidence is only realistically likely to be considered if it chimes with any prevailing ideology. But approach and method will often matter too. As you will know in the United States, for example, the 2001 Elementary and Secondary Education Act (No Child Left Behind) has led to a preference and increased funding, for randomised controlled field trials. In the UK there is also a preference for RCTs, and meta analyses.
Now, where can interested individuals, reading specialists get a hold of this book?
Chris Brown’s book, Making Evidence Matter: A new perspective for evidence-informed policy making in education, ISBN 978-1-78277-020-6 is aimed at both educational researchers and policy makers interested in enhancing the use of research outputs in policy making. While the book’s primary domain is education, the concept of evidence-informed policy making has salience across a number of sectors, notably health and social care.
It is available to order in North America from Stylus Publishing www.styluspub.com and can also be ordered from online book retailers such as Amazon and library distributors. In the UK you can purchase it from all good bookshops, online retailers and library distributors. Please visit ioe.ac.uk/ioepress for more book information and suggested ways to order.
The Institute of Education is a college of the University of London that specializes in education and related areas of social science and professional practice. In the most recent Research Assessment Exercise two-thirds of the Institute’s research activity was judged to be internationally significant and over a third was judged to be “world leading”. The IOE was recognized by Ofsted in 2010 for its “high quality” initial teacher training programmes that inspire its students “to want to be outstanding teachers”. The IOE is a member of the 1994 Group, which brings together 19 internationally renowned, research-intensive UK universities. The Group was established in 1994 to promote excellence in university research and teaching. www.ioe.ac.uk