Be Careful… ChatGPT Appears to be Making up Academic References

Is ChatGPT making up references?

[00:00:00] David: Is chat G P T, making academic references. Hi, I’m David Wilkinson. I’m the editor of the Oxford Review and an interesting thing happened the other day. One of my students turned in a project and I was looking through the references and there was one that I didn’t recognize. Uh, so I went and checked it out.

[00:00:19] I did a quick Google Scholar and I couldn’t find it. I went to the actual, Journal and I couldn’t find it and I was thinking, this is a little bit weird. So I started with chat G P T to see if it had been generated by there. And guess what? So what I’m gonna show you now is chat, G P T and how it generated.

[00:00:40] A reference and I went to chat G p T, so I put in some search terms. This is part of a conversation that I was having, and firstly, it references a 2013 Harvard Business Review article, which I’m not interested in largely because they’re not peer reviewed. Um, And quite a lot of them are opinion pieces [00:01:00] rather than research articles.

[00:01:01] In fact, I don’t know of any primary research articles that are ever published in Harvard Business Review, but that’s by the by. So here I find something that sounds similar to. The research that my student referenced here.

[00:01:16] ChatGPT: Similarly, a 2017 study published in the Journal of Leadership Education found that many leadership development programs do not adequately incorporate research-based best

[00:01:26] David: practices.

[00:01:27] So I decided to ask a little bit more. What is the 2017 study you referenced? And it started giving all of this detail about where it was done and who did it and what the main findings were and everything. So I asked them to list the best practices that were in the thing, and it did that, and I’m not going to read through all of that.

[00:01:46] And then I asked, give me the reference, and they said,

[00:01:48] ChatGPT: sure. Here is the full reference for the 2017. Klein, h j Nowman, s e, and Walnut J 2017, [00:02:00] leadership development programs and evidence-based review. Journal of Leadership Education 16 1 273 to 285. So

[00:02:11] David: what we now need to do is go and see if we can find this study.

[00:02:15] So the first thing I’m going to do, the easiest thing is just to copy this bit here and we’ll go to Google Scholar and see if we can find it in Google Scholar. Copy that in and see what. Now that’s a ominous sign because if they found an exact match, it would’ve come up and that would’ve been the only paper there.

[00:02:34] So let me just go back to chat, g p t here and see who we’re talking about. Climb Newman and Walner. Interesting. Again, I’ve not heard of them. So, um, we’ll do that and see whether that comes up with something.

[00:02:54] No, nothing at all. Now, that doesn’t mean anything because Google Scholar can get things [00:03:00] wrong too. So what we’re gonna do now is see if we can hunt down the journal and see whether it’s in there. So they’re quoting the Journal of Leadership Education, so let’s go and find that. , see what that says. Here we go.

[00:03:12] Journal of Leadership Education. That was the one, and it was talked about 2000, um, 17, and it was saying that it was volume 16, number one. Let’s do a search first based on the title that we’ve been given and see if we can find it, because it may have just been misfiled

[00:03:36] nothing. No, none of those are even similar. So let’s just go to the issues. And it was volume 16, 2017. So let’s go in there. And it was issue one. So let’s go and have a look at what’s in issue one.

[00:03:57] No different people. So that’s not the same article. [00:04:00] Now research. Let’s have a little look at this. Because it was suggesting it was research. No, no. No, no, no, no, no, no. No, none of those. So it wasn’t even in that issue. The other thing that it gave us was a doi. Now the DOI is a kind of a international reference number for research papers.

[00:04:26] So let’s just go to, um, the doi. So we’ll just put the DOI into the official DOI website and see what it comes up. Now this is a really weird one. They’re not the kind of numbers that I’ve seen before, but no, not found, so it doesn’t seem to exist. Let’s just shorten that and see if we can find something just from those.

[00:04:49] All right, so we’ve got one journal of leadership education, so that’s obviously just for the journal. The paper doesn’t exist. Well, I’ll tell you what, let’s just go and do a Google search, [00:05:00] put in. Oh, well, let’s do that first. See whether that comes up with anything. Yeah, well, it’s the same one that’s interesting cultivating leadership development, but they’re different people.

[00:05:13] Let’s just see whether No completely different and that is different as well. Let’s take the title and see what we’ve got.

[00:05:27] No, Evan spoke no medical leadership development. That’s a different one. And a different author. Different author. Different author. There’s nothing. So what are we faced with here? Well, there’s a couple of possibilities. One, You’ve gotta remember that chat. G P T is generative it. It’s generating text. So it could have made it up, I don’t know.

[00:05:50] But it could also be that it’s found the reference in something else. And what it’s done is it’s generated the text mixed in with some of the things as well, because what it was [00:06:00] actually talking about. So if we go back to what chat g p T was saying, it looks, it sounds really legitimate, you know, um, it’s saying these are the findings and I’ve searched for these as well.

[00:06:12] I can’t find this paper is chat G P T generating academic references as well. I have no idea whether it was a reference that some student has put in that they’ve just made up. I dunno where that’s come from, um, but I certainly can’t find it. And that paper doesn’t seem to exist. So you need to be very careful when using chat G P T and assuming that the references that you’ve got or that the paper that it’s showing you is actually right as they say caviar, empt, or.

[00:06:44] you know, just because it’s coming out of chat, g p t doesn’t mean it’s right. It seems to be making stuff up or it’s copying stuff from somewhere that’s not correct. You need to be very careful about what you’re doing, particularly in academic circumstances, but also professional [00:07:00] circumstances. So if you’re doing a.

[00:07:02] Uh, presentation or something and you reference something that you’ve got from chat g p t and you haven’t done your homework may well be wrong and this isn’t the only case that I’ve had of, uh, references that chat. G p t have ge generated that when I’ve gone in and had a look at them. Either the reference wasn’t a lot right or what chat g p T says the paper was saying is also not right.

[00:07:24] Be what?