Wiki Research talk on hoax articles and Wikipedia election prediction: transcript

Key points

• The talk picks research papers from the year on information accuracy and content quality on Wikipedia-related projects, as fake news became a major public debate.
• One paper studies about 21,000 articles removed as hoaxes on English Wikipedia, comparing them with articles flagged as hoaxes that turned out to be legitimate.
• Hoaxes that survived used more wiki markup relative to plain text, probably to look like legitimate articles.
• Only about 1% of hoaxes survive a year, about 1 in 20 last more than a day, and most are gone within an hour.
• The machine classifier was right about 90% of the time, against about 66% for a recruited Mechanical Turk worker.
• A second paper by two UC Santa Barbara PhD students uses page views of US Senate candidates in 2008, 2010 and 2012 to predict election results.

Chapters

1. 0:00Research on hoax articles on English Wikipedia
2. 3:08Wiki markup ratio in hoaxes versus legitimate articles
3. 4:50Hoax survival rates and classifier accuracy
4. 6:21Predicting elections from Wikipedia page views

Full transcript

0:00We're going to be talking about this because for most people in this space, the research on this is just starting to grow, especially as fake news, not just information accuracy, which sounds a whole lot less interesting, has become a major topic of public debate, at least in the United States, if not elsewhere. We picked out of the papers this year that talk about information accuracy in relation to the quality of content on Wikipedia-related projects. We picked out one, which is this paper here, that really talks about hoaxes and hoax articles on English-language Wikipedia, in this case.

0:38The authors for this paper, which includes the lead author, I believe is a grad student at the University of Maryland, Srijan Kumar, and the other two authors are both at Stanford, Bob West and Yuri Leskovich, talks about the set of articles that have ever been flagged as a hoax on the English-language Wikipedia, and focuses most of its analysis on the roughly 21,000 or so articles that are subsequently removed as hoaxes. So they look at these because they want to understand a few different things.

1:11They want to understand what sorts of features distinguish articles that turn out to be taken down later as hoaxes from articles that... Only that somebody along the way thought might be a hoax, but turned out to be legitimate. They want to understand how those articles are edited differently, how the content differs, they want to understand how frequently they get viewed, and how long they survive on the wiki. And finally, a really interesting thing that they tried to do in this article is that they built a machine classifier, so a machine learning system, that would try to classify automatically whether an article was a hoax or not.

1:49And then they tested the performance of that classifier to see how accurately it could do that. And they also compared that against a human classifier, basically a person who they recruited from Amazon Mechanical Turk, so not a Wikimedia. I'm sure all of you can classify, hoaxes or not, with 100% accuracy. They recruited a mere mortal to try to see how they would perform in comparison to their machine learning classifier. In keeping with our general pattern here, just trying to pick out one interesting result, you know, this paper is filled with interesting points about hoax articles and lots of great figures and lots of great tables, so if you're interested in this, I really encourage you to check it out.

2:30It was also summarized briefly in the, I think, October newsletter? The computer research newsletter, so you should check that out, too. But I just picked out a few things, so don't worry. I know you can't really see what's up there. Let me just draw your attention to like a couple of cool things, and then I'll read off some others that I've got for you now. So one thing that's interesting, let's just start in the top right there. If you can see my little gray pointer, my green pointer, I'll try to circle around here. So you can't really tell, but the interesting lines here are this kind of pinkish, reddish one, and the yellow one.

3:08So, let me just make sure I'm reading that right. Yeah, so the pinkish-reddish one corresponds to articles which somebody flagged as a hoax, which turned out later not to be a hoax, right? So after getting flagged, other people came in and talked about it. It turned out, okay, this article is legit. We're going to leave it. And the yellow line corresponds to articles that were hoaxes that actually survived for a little while on English-language Wikipedia, right?

3:40And what we're looking at here along the bottom axis is a ratio of how much plain text versus wiki markup it is, right? So the further out along this axis you are, you have more text and less wiki markup. You with me? The closer you get to the zero point here along this axis, the more wiki markup you have versus plain text. So, if you were to show up at an article and see that it had only plain text, and you were suspicious about the content of this article, seeing only plain text, because we think of good articles on Wikipedia as having a markup, right?

4:19So they've been turned into carefully crafted wiki markup articles with things like info boxes and templates and all this jazz. But what you actually see in this case is that articles that turned out to be legit are further out on that axis than the articles that turned out to be hoaxes. So what that means is that the people who were successful at putting hoaxes on English-language Wikipedia were adding more Wiki markup, probably in an effort to help disguise their hoaxes as legitimate articles. And there are a few others like this that are kind of interesting.

4:50So it's a tiny proportion of the hoax articles that actually survive. Only about 1% make it a year, and only about 1 in 20 make it more than a day. Usually they're gone within an hour. But of those articles that turn out to be hoaxes, they tend to kind of have more of the features that look like good articles, infoboxes, templates, wiki, markup, than the articles that are wrongly flagged. Right, so this is sort of a nice little mind-blowing finding that I think is really cool, and like I said, I totally encourage you to go check this paper out because you can also learn about what they did to test their classifier. Their classifier, it turned out, was really accurate at detecting hoaxes. The classifier was able to look at an article and say, this is a hoax or not, and be right about 90% of the time, and when they line that up against a human they recruited for Mechanical Turk, the Mechanical Turk worker was only able to classify it right about 66% of the time.

5:47So, it's really interesting to think about what, one, what features they found that were distinguishing true hoaxes from not true hoaxes, that's a complicated idea, but then also interesting to think about how that might be useful either as incorporating it into editor workflows or as a way to start thinking about what features we can use to identify good faith contributions that maybe raise suspicions about hoaxes so that we can do more to not scare those people off by calling them liars and trolls and fake news.

6:21Okay, on to the next thing. So this is another area of research that's been booming along in a nice little clip for the last few years, which is using Wikipedia data of various kinds to try to predict certain things in other parts of the world. So, you can find examples of this that are trying to predict, right, that are kind of placing Wikipedia data head-to-head against things like Google Flu Trends, although that didn't work out so well, it turns out. It turns out that there's better data out there if you want to understand where the flu is going to go. Wikipedia's not so helpful.

6:57But we found one article this year that seemed pretty cool and interesting, and for some reason all the ones I'm talking about have to do with politics, so apologies for that. But this article is done by two PhD students at UC Santa Barbara in communication, which I like because that's field I'm in, so it's great to see graduate students in the field that I'm in doing interesting work. And what they did was they tried to use page view data to predict electoral outcomes. And what they did was they gathered pages about senatorial candidates in the United States in 2008, 10, and 12 in English language Wikipedia.

7:35Again, both of these papers about English language Wikipedia, which is pretty common in a lot of this research field. So these folks gathered data, gathered pay-tree data for senatorial candidates in the United States in three years. And they used that to compare it against data from opinion.

Source and licence

Recording: File:Wikimedia 2017 Wiki Research talk on Fake News.webm by Jwslubbock (2017-08-11), via Wikimedia Commons. Licence: CC BY-SA 4.0. The transcript, summary and chapters on this page are shared under the same licence (CC BY-SA 4.0).

The transcript is machine speech-to-text by the TidyTools Audio & video transcriber; the summary, key points and chapters were written by Claude from that transcript. Both may contain errors: check the recording before quoting it.