Podcast – how some researchers reduce trust in research

Source manipulation

• Subscribe on iTunes
• Player FM
• Listen on Spotify
• Listen on Podbean

Transcript

(00:00):

Why do some people do this? How some researchers reduce trust in research. Hi, I’m David Wilkinson, the editor of the Oxford Review. I came across this last week and it’s both blatant and appalling because it really reduces people’s trust in research. Now you may have come across this concept here, it’s called the learning pyramid. Basically what it’s saying is it came out of the thing called the National Training Laboratories in the late 1980s. Basically what it’s saying at the top of the pyramid, the smallest part, is saying that students retain about 5% of everything that’s coming out of a lecture, about 10% from reading, about 20% from audio visual or video. They retain about 30% from watching a demonstration. They retain about 50% from group discussions, about 70% from practicing something, and 90% right at the bottom of the pyramid, they retain about 90% from teaching others.

(01:00):

I’ll say, you may have come across something like this before. It’s been around for a very, very long time. It doesn’t mean that it’s true by the way, which we’ll come onto in a minute. So last week I’m reading a research paper and this is it. Developing Management Effectiveness: The Nexus Between Teaching and Coaching. It was published in the International Journal of Management Education just last month.

Now the paper, which I was surprised about actually references the learning pyramid. It uses a reference here called [Latrud 00:01:36] and it says adapted from Latrod. Now, the reason I was surprised about this was that as far as I’m aware, this isn’t true and it’s not true for a number of reasons. One is there are so many variables involved in learning. Things like depending on your expertise, your prior knowledge, whether you’ve already got a framework of thinking about the subject, whether you’re a good critical thinker or not. Reading can be every bit as effective as teaching somebody else.

(02:01):

In fact, teaching somebody else has actually been shown not to be as effective as some other things again, depending on the circumstances. So I’m intrigued now as to why somebody would be using this and also this reference of being adapted from Latrud. So let’s just go on and see what the paper’s saying about this. It says the complimentary nexus between classroom-based management education and coaching is however, and this is an important part, most strongly reinforced by studies conducted by the National Training Laboratory or NTL in Bethel, Maine US in the late 1980s.

(02:42):

So most strongly reinforced by studies, which is suggesting that there are some studies. The NTL studies empirically reveal the notion of a learning pyramid. That suggests that there are some experiments. They’ve done experimentation and this is right based on those experiments. In that learning is retained at graduated stages of learning interventions. So just like the pyramid’s saying that students tend to learn or retain less from attending a lecture or reading than they do from teaching this stuff. They’re using as a reference here of Latrod, that’s a person’s name based on a paper that was published in 2012.

(03:30):

So just thinking about that, just remember what they’ve said, the NTL studies empirically reveal the notion of a learning pyramid. So when I go to the bottom of the paper and look at the reference, this is what I find.

Latrud, A Rebuttal of NTL’s Institutes Learning Pyramid. Now I’m confused because that doesn’t sound like just from the title, like empirically revealed the notion of a learning pyramid, that it’s evidence of it. It sounds like the opposite. So anyway, let’s go and have a look at the paper and see, because titles could be anything.

So I’ve got the paper out and this is what it says. The article discusses the learning pyramid propagated by NTL. It presents and compliments the historical methodological critique. What that means is lots of other people have said this is not true, right? There’s lots of reasons for it. Against the learning pyramid and calls for the NTL Institute to retract their model.

(04:27):

Now, for researchers to actually put that on a paper means that there’s very good evidence for saying that. It’s a very unusual thing to do. Bearing in mind what they’ve said is the NTL studies in particular reveal the notion that this is saying the opposite. They’ve used this as the reference for backing up the assertion that the learning pyramid is true when blatantly it isn’t. So let’s just have a quick look at the conclusions of Latrud’s paper.

The NTL’s Learning Institute lacks empirical evidence, the exact opposite of what that paper’s saying. Any attempt to perform empirical tests of the model will encounter methodological problems. As I said, there’s too many variables. Despite its inability to stand up to close examination, the model is still corroborated by the NTL Institute and continued distribution of this model cannot be justified. In order to reduce further dissemination, the NTL Institute ought to retract it.

(05:26):

That can’t get much less ambiguous than that. That’s pretty unambiguous statement. Yet the paper that we looked at is saying the studies, Latrod’s saying the exact opposite of what they’re saying Latrod is saying. You couldn’t get any more blatant. There’s no mistaking what Latrod’s saying. There’s no ambiguity in any of this. What I’m confused about is this. Why do people do this? I think there may be a couple of reasons and my thoughts are, and these are the kinds of practices that are see in the review when we’re reviewing papers because we probably get rid of around about 40% because of either the evidence is lacking or it’s poor studies or something. That’s on a good day. On a bad day, it can be up to 80% of the research that we read. I think there are a number of reasons why this occurs.

(06:22):

I think the first reason is that people kind of make up a theory based on their intuition. What they try to do is they try to dress it up as research or retrofit evidence from other places. I think that a lot of it is just poor quality research. So you get people doing things like they’ll publish a study based on 10 people in my office, which is hardly representative, right? So we get a lot of that. Or the evidence is just inconvenient. Therefore, what we’re going to do is we’re going to manipulate it and present it as something that it isn’t. That’s blatant and lying basically. Or just sheer laziness. There’s quite a lot of laziness goes on in, well, just about every aspect of work. Now, obviously the basis of this is if you’re reading research, check the sources because they may not be saying what you think they’re saying. What I’m interested in is what do you think about this?

(07:19):

What I’d like you to do is just put what you think and feel about this kind of practice. Why do you think people are actually doing it? Why are researchers doing this? It’s nuts and it’s pretty blatant and obvious. Anyway, I’m David Wilkinson, editor of the Oxford Review. If you want to be the most well-informed person in the room, go to Oxford-review.com and you can sign up for free research briefings and a whole load more. Okay. To become the most well-informed person in the room about people and organizations, including topics like leadership, management, organizational development, organizational change, human resource, human resource management, learning, organizational learning, learning and development, coaching, and work psychology and decision-making, go to Oxford-review.com.