Good evidence

The purpose of science is to reduce uncertainty. That is not to say that science claims to produce 100% certainty, but it aims to explain things in a way that makes the world more predictable, constructing a more solid reality than if we just guessed at explanations. It’s generally assumed that there’s one proper scientific method and it’s what scientists do, standing at their laboratory benches in their white coats. It might come as a surprise to some people that there are scientists and sciences of different types, depending which bit of reality is under their consideration. It’s an important aspect of science’s work is to investigate itself – to be reflective about which type of science might increase certainty in any particular reality.

I’ve read Dr Ben Goldacre’s recent article for Mr Gove on evidence based practice in education (http://www.badscience.net/2013/03/heres-my-paper-on-evidence-and-teaching-for-the-education-minister/ ) prompted by a email note from Tim Taylor .

I’ve mulled this over for several days, looked at the responses to the Guardian item by Dr Goldacre and a few other blogs and there are a couple of questions I want to address.

There are two parts to my argument. The first is about the scientific investigation itself and the second is about the application and effect of the results of the investigation.

Randomised controlled trials, projecting the science of numbers into the lives of people, have their place and Dr Goldacre’s project to strengthen the role of good science in medical decision making is admirable. At the same time it’s generally true that what gets measured is what’s measurable. That means if something can’t be translated to numbers it won’t be the subject of a good randomised controlled trial. To set up a randomised controlled trial the researcher makes a guess as to what the outcome of the trial might be and what might be the opposite of this outcome; the experimental hypothesis and the null hypothesis. The experimental variable, the assumed cause of the effect is being guessed at, must be measurable by an instrument of some kind; thermometer, pressure gauge, people counter, whatever.

For example in his book ‘Bad science’ Dr Goldacre critiques the evidence for homoeopathy. He makes a strong scientific case for the removal of homoeopathic remedies as evidence based medicines, because the active agent which mimics the symptoms of the disease in question is absent from the treatment. Certainly the dilution levels of the potentially active disease mimicking agent are so great that at a molecular level the original agent is effectively and statistically absent from the final preparation.

However Dr Goldacre also talks in the book about the well-known placebo effect. There is evidence related to this effect that he quotes in his book; and the recasting of the ‘placebo’ effect as the ‘meaning’ effect. If there is meaning, without molecules, it seems that the cure might just work. In the mid-20th century Viktor Frankl came up with the idea that the meaning of an intervention is significant and developed his ideas into the school of Logotherapy in Eastern Europe.

Whilst I respect Dr Goldacre’s work within his frame of reference, he’s making a common mistake in overclaiming the application of positivist scientific methods to sociological phenomena.

He talks a bit about interventions designed to change phenomena in the social world and the measurement of such changes. An intervention in his terms is a cause intended to bring about a particular effect, conforming the experimental hypothesis. In the reality of the medical world in which he’s operating, there are physical mechanisms operating, for example this exercise strengthens that muscle or this molecule inhibits that enzyme. However this reality does not allow for agency, a person’s ability to influence the world.

In the laboratory I can set up a system to investigate the strength of 8 year old children’s arms, the children having been given a course of specified arm strengthening exercises in school, by trained trainers. The experimental group will have the exercises and a matched control group will have no specific exercise programme. Having read other published work on exercise and outcomes, my experimental hypothesis is that ‘the programme of arm exercises will produce a 20% increase in arm strength in the experimental group.’ I will need a large experimental group to be able to carry out statistical analysis, so 2000 children do the exercise programme. I also identify 2000 children matched for age and gender who will get no specific exercises. They are the control group. Everything is ready to go, the first groups are coming into the lab where we will measure their strength. The subjects of the experiment are the children’s muscle’s. The fact that some children will drop out of the trial or won’t cooperate with the trainer is provided for in the experimental design. The children themselves are means of getting their physiology into the lab. The children are necessarily and properly reduced to numbers.

In doing sociological science to investigate social reality, the objects of investigation are the people themselves. Methods are designed accommodate the fact that the people who are the experimental subjects have agency, they can and do make choices. This needs a different form of science to investigate that reality, social scientific methods to investigate social phenomena.

What Dr Goldacre has missed out , in common with many investigators, is the ontological question; does this science match that reality?

In his commissioned article he was addressing a journalist, Mr. Gove and a venture capitalist, Lord Nash. They were hardly likely to raise the ontological question because they probably don’t know it exists. It’s evidenced by the internal inconsistency shown by Mr Gove, proclaiming the importance of evidence at the same time as ignoring it. The Education Minister is putting the training of teachers in the hands of Teaching Schools, modelled on teaching hospitals. Where is the evidence that teaching hospitals can produce high quality teachers? Schools aren’t hospitals and teachers are not medics. Their realities are different. The confusion is shown up by Mr. Gove’s 2010 policy directive:

‘We will reform initial teacher training so that it focuses on what is really important.

    1. The initial training of teachers is perhaps the most important part of their professional development. Over a twenty year period, initial teacher training has tended to focus more sharply on classroom practice. Even so new teachers report that they are not always confident about some key skills that they need as teachers, for example the teaching of systematic synthetic phonics as the proven best way to teach early reading and the management of poor behaviour in the classroom.

Synthetic phonics is a mechanical, skills approach to reading, it’s does not take into account the learner’s agency, everyone has to do it. As a cause-effect pair, synthetic phonics/reading, it would be susceptible to assessment by randomised controlled trial, if there was an agreed meaning of the term ‘reading’. The evidence that Mr. Gove has seen has convinced him personally that it is ‘the proven best way’.

Mr. Taylor, the Government’s ‘behaviour expert’, casts the management of children’s poor behaviour as another cause-effect pair; recite the schools discipline rules twice a day/children will behave. He claims that the evidence shows his method works. He’s no scientist but he’s making the same mistake as Dr. Goldacre. Rote learning to read is a mechanical phenomenon; children’s learning to become themselves as people is a social phenomenon. Hard outcomes and soft outcomes. Different reality, different science.

The second part to my argument is about the application of research findings, and in my opinion this is the biggest barrier to the useful deployment of large scale randomised controlled trials in education. Whatever methods are used to construct and conduct experiments in medicine, if a trial demonstrates that an intervention achieves results that are as good as, or better than that which is already available, the intervention can be assessed by the National Institute for Clinical Excellence and if approved will be delivered in a consistent way by medical professionals. The delivery can be monitored for consistency and the results monitored for effect and cost.

However, whatever methods are used to carry out research in education the application of the intervention is always a social/cultural act. Medical professionals have a common belief system which overcomes personal prejudice and individual experience. At least that is the official position. Educational professionals engage their experience, personal prejudices, creativity and other characteristics in their pedagogy – the art and science of teaching. This is not what medical professionals do. I would suggest that whatever randomised controlled experimental findings might be, there would be no overall standardised process of intervention, or what we’d call pedagogy, to ensure consistent application.

It’s been tried already, it usually doesn’t result in long term sustained improvement.

If science really is about increasing certainty then we should be looking to what science can do this, whether it’s extensive randomised controlled trial, ethnography, intensive action research, thought experiments or the reflective cycle of the professional. Critical realism or scientific positivism?

Both/and, not either/or.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *