How can citizens learn to fact-check the news online? To find out, NewsTrust created a new service called Truthsquad and invited our community to fact-check controversial quotes from politicians and pundits, with professional guidance. (see earlier blog post)
Our first pilot ran for a week, from August 2nd to 8th, 2010, with the help of our partners at the Poynter Institute and our advisors at FactCheck.org -- and with the generous support of Omidyar Network and the MacArthur Foundation.
Our Truthsquad editors included: Kelly McBride, director of Poynter's Sense-Making Project; Steve Myers, managing editor for Poynter Online; Fabrice Florin, NewsTrust's executive director; and Jon Mitchell, our new managing editor. Throughout the pilot, Brooks Jackson, director of FactCheck.org provided helpful tips and advice.
Here are our findings from this experiment in community fact-checking.
Overview
We created Truthsquad to help expose misinformation on the Internet -- and to give people new tools to fact-check the news, with professional oversight. Our hybrid 'pro-am' approach leverages the expertise of a few experienced journalists to show citizens how to tell fact from fiction in the news.
Each day during our pilot, our editors featured new quotes to research, and asked our community whether they were true or false. These short quotes were excerpted from recent news reports, opinions or political ads suggested by our editors and advisors.
Each quote page included links to factual evidence supporting or opposing that quote. We invited our community to review these links before giving a final answer. Editors actively searched for (and linked to) new evidence for their assigned quotes, monitored the community's answers and wrote a verdict based on the factual evidence we dug up together. We also held lively discussions of the quotes in our comments sections.
In all, we fact-checked twelve quotes as a community, with 528 answers from 307 participants. Each participant gave answers of "True," "False," or "Not Sure," with an average of 44 total answers per quote. Together, we posted 132 links to related evidence, which received 241 NewsTrust reviews. Participants were welcome to change their answers at any time, if they came across new factual evidence. As a result, participants changed their answers 57 times (which we view as a positive sign).
Truthsquad has generated more interest than any of the other pilots we hosted in 2010, and web traffic was also higher than usual. Over the week, we logged 12,801 pageviews for Truthsquad, and we estimate that at least 3,723 unique visitors checked it out, whether they participated or not.
In general, our community's responses were consistent with verdicts from our four Truthsquad editors. We were also pleased that comments from participants were civil, and that they seemed genuinely engaged in this communal quest for credible information.
Check it out for yourself. If you haven't already, try to fact-check some of the quotes on the Truthsquad overview page. Our final verdicts are posted on our earlier blog post (as well as on each quote page).
Based on this favorable response, we would like to offer Truthsquad on an ongoing basis, and we are now seeking donations and foundation support to provide it as a regular feature of our news curation service.
Report Contents:
• Quotes
• Featured Quotes
• Stats
• Survey Responses
• Thanks
• Next Steps
Related Stories about Truthsquad:
• GigaOm's Article on NYTimes.com
• Kelly McBride's Report on Poynter
Quotes
Here are the twelve quotes we fact-checked for this Truthsquad:
- "87 million Americans will be forced out of their coverage" by Obama's health care plan
By Orrin Hatch, U.S. Senator - 119 answers (featured below)
- Decline in illegal immigration to U.S. "due to the recession," not enforcement
By Dick Morris, Fox News consultant - 105 answers (featured below)
- "Crime is totally out of control" in Phoenix
By Bill O'Reilly, Fox News - 86 answers (featured below)
- "Seniors guaranteed Medicare benefits will remain the same" under the Affordable Care Act
By Kathleen Sebelius, Secretary of Health and Human Services - 67 answers (featured below)
- "The 'Bush tax cuts' ... remain the single largest cause of America's structural deficit."
By Fareed Zakaria, Washington Post - 65 answers (featured below)
- Environmental impact of the oil spill "much less than everyone feared."
By Jacqueline Michel, Geochemist - 33 answers (featured below)
- "CIA and Google investing in "a company that monitors the web... to predict the future."
By Noah Shachtman, Wired - 19 answers
- "Warren Buffett called Greene's scheme 'financial weapons of mass destruction.'"
By Kendrick Meek (D-FL), U.S. Senate candidate - 17 answers
- "There's no evidence of any significant concentration of oil that we haven't accounted for."
By Jane Lubchenco, head of the National Oceanic and Atmospheric Administration - 15 answers
- "Meek lobbied for big tobacco against children's healthcare."
By Jeff Greene (D-FL), U.S. Senate candidate - 14 answers
- "Norton pushed the largest tax hike in Colorado history."
By Americans For Job Security - 12 answers
- Obama's choice for budget chief "received more than $900,000 from Citigroup"
By Jim McElhatton, The Washington Times - 9 answers
The first six quotes above are featured below, with more findings. To see our verdicts for all twelve quotes, check our
earlier blog post.
Featured Quotes
Here are more detailed findings on the six quotes which we featured during this pilot.
"87 million Americans will be forced out of their coverage"
by Obama's health care plan
By Orrin Hatch, U.S. Senator
Our verdict: FALSE
119 answers
Our investigation of this quote from Sen. Orrin Hatch led to some of the best collaborative fact-checking in this pilot. With 123 answers, this quote was the most active of the week, and 105 of these answers agreed with our verdict that Senator Hatch's statement was "False" (eight said "True," and ten were "Not Sure"). This strong consensus may be due to the great teamwork of
Truthsquad editor
Steve Myers and NewsTrust member
Gerald Zuckier, who jointly dug up the actual text of the health care regulations and helped produce an impressive verdict.
From Steve Myers' verdict: (on behalf of Truthsquad editors)
"This quote wasn't easy to fact-check, for several reasons: Hatch doesn't provide a source for his figures or much of a clue as to what he's talking about. A Republican member of Congress issued similar talking points that provided an incorrect link to the federal regulations he's referring to. And while Hatch says this with certainty, the regulations only predict what could happen in the future. (...)
Score one for crowdsourcing, though: NewsTrust contributor Gerald Zuckier found the right set of regulations, read them and concluded that Hatch misstates what the regulations say. After reading the regulations, we agree."
See full verdict »
Decline in illegal immigration to U.S. "due to the recession,"
not enforcement
By Dick Morris, Fox News consultant
Our verdict: MOSTLY FALSE
105 answers
For the two weeks before this pilot, we hosted an Immigration News Hunt with USA Today, so we were well prepared to research this quote about illegal immigrants. Morris' statement proved controversial; out of the 107 respondents, 54 found it false, while 36 answered true.
Contributing edior Derek Hawkins, after consulting the linked sources, concluded that "enforcement appears to have played at least as great a role" as economics, answering "False." Randolph Selig found the statement "True," contending that "the big draw is jobs." Still others, such as TC Reg, were "Not Sure." Reg thought that "it's impossible to say for sure as no scientifically valid studies have been conducted on the subject."
Eventually, the Truthsquad editors ruled that the statement was "Mostly False." The verdict was based on two major studies of immigration patterns on the U.S./Mexico border from credible sources. The linked sources that opposed the quote received an average NewsTrust rating of 3.4, slightly higher than the 3.1 average for sources that supported Morris.
From Jon Mitchell's verdict: (on behalf of Truthsquad editors)
"Downward economic trends... must have had an effect, but both studies point out that rates of legal immigration remained constant as illegal immigration dropped. Though illegal immigration comprises most of the traffic, and economics are clearly a factor, the constant rate of legal immigration suggests that some other factor makes it harder for people to enter the country illegally."
See full verdict »
"In the capital city Phoenix, crime is totally out of control."
By Bill O'Reilly, Fox News
Our verdict: FALSE
86 answers
Our members posted evidence from a wide range of sources that disproved this quote. While some other participants posted links to support O'Reilly's assertion, the group found those far less trustworthy. The sources that opposed O'Reilly's statement received an average NewsTrust rating of 3.7, a strong indication of credibility, whereas the sources that supported O'Reilly only merited a 2.6 rating.
At first, it seemed that this statement might be too subjective to fact-check. "Out of control" is not an easily measurable quality. However, the data showed that crime in Phoenix has been dropping for several years, a trend which the editors felt was sufficient to debunk O'Reilly's claim.
From Jon Mitchell's verdict: (on behalf of Truthsquad editors)
This statement would have been difficult to fact-check, if the data had been more ambiguous. O'Reilly's statement that crime is "out of control" would ordinarily be open to subjective interpretation. However, the only sources that corroborated O'Reilly's statement seemed clearly biased, and trustworthy sources, including the Phoenix Police Department and the FBI, indicated that crime in Phoenix has been dropping steadily for years. ... Bill O'Reilly's assertion that crime in Phoenix is "out of control" would require at least a measurable increase in crime to be considered factual."
See full verdict »
"Seniors guaranteed Medicare benefits will remain the same"
under the Affordable Care Act.
By Kathleen Sebelius, U.S. Secretary of Health and Human Services
Our verdict: HALF TRUE
67 answers
Links to some related articles suggested that Sebelius' statement was technically true, and many of our participants agreed. Jim Lang wrote that "the term 'guaranteed Medicare benefits' allows it to be true." Indeed, no Obama administration spokesperson we came across discussed this point without appending the word "guaranteed," which editor Kelly McBride considered "slippery." Lynn Caporale clarified the meaning of this buzzword, pointing out that only "the extra subsidies to the Medicare Advantage plans" will be cut, which Barry Parr contends "are clearly not 'Medicare' under any reasonable definition." Thanks to the word "guaranteed," many people felt Sebelius was in fair territory. But other members rejected the administration's claims entirely. Morriss Moore wrote that "the health plan will limit health care, second opinions, and will have 'panels' to determine the amount of care one receives."
Due to the lack of clarity in Sebelius' wording, Truthsquad editors could not consider the statement to be true. Fabrice Florin wrote in his verdict on behalf of editors that "because this quote is partly misleading, we give it a rating of 'Half True'. We think the Obama administration could have been clearer and more forthcoming on this point."
See full verdict »
"The 'Bush tax cuts' ... remain the single largest cause
of America's structural deficit."
By Fareed Zakaria, Washington Post
Our verdict: TRUE
65 answers
This quote proved contentious, and some members vehemently disagreed with 48 who voted "True." Zakaria's claim is not a neat statement of fact. The tax cuts may be the largest component of the deficit, but that does not necessarily make them the cause.
As a result, answers to this Truthsquad seemed to be partly influenced by the political views of some participants. For Tyson Emmett, the operative word in Zakaria's statement was "cause." He argued, "Spending causes deficits. The underlining cause of every deficit is needing more than you have." On the other hand, Chris Strosser asked, "How can Republicans complain about deficit when the Bush tax cuts are the largest chunk?"
Rather than inflame a political debate, the editors focused on the data. A report from the Center on Budget and Policy Priorities showed that, if the loss in revenue due to the tax cuts is measured as a cost, it dwarfs all other structural programs in the budget, including the wars in Iraq and Afghanistan. Since the tax cuts are an elective part of the budget, the judges concluded that Zakaria's statement is "True."
See full verdict »
The environmental impacts of the BP oil spill "have been
much less than everyone feared."
By Jacqueline Michel, Geochemist
Our verdict: MOSTLY TRUE
33 answers
This was one of the hardest quotes to fact-check, because it is not a statement of fact, but "an opinion about an opinion." Many participants pointed out the ambiguity of this quote, and over a third of the 33 respondents selected "Not Sure." In their answers, people highlighted three major reasons for withholding a verdict. Eric Yendall succinctly summarized one with a question: "Who exactly is 'everyone?'" Another problem with the statement is that, as Steve Myers pointed out, "it's too early to tell." Finally, in Joey Baker's words, "we have no idea what impacts everyone feared."
However, after reviewing the evidence, the editors declared a verdict of "Mostly True," accounting for the ambiguity of the statement, acknowleging that the outcome is not yet certain, but pointing to a range of scientific opinions that damage from the spill has been less than expected, compared to previous undersea oil disasters.
From Kelly McBride's verdict: (on behalf of Truthsquad editors)
"It appears true that many scientists and other experts say out loud that they are pleasantly surprised by the resiliency of the gulf, the beaches and the wetlands. There have been hundreds of oil-soaked birds, not thousands. Oil on the beaches has washed up in much smaller amounts than predicted. Very few injured or dead mammals have been recovered. The oil appears to be breaking down and evaporating more quickly that was thought possible. ... So the environmental impact of the spill is in fact not as bad - at this point in time - as experts thought it would be. However, almost every respectable expert qualifies that statement by cautioning a wait-and-see attitude."
See full verdict »
Stats
Here are some of the quantitative results of the weeklong Truthsquad pilot, from Aug. 2 to Aug. 8th, 2010.
Traffic:
Participation totals:
- Total participants: 307
- Total answers: 528
- Total links posted: 132
- Total links viewed: 572
- Total links reviewed: 241
- Total changed answers: 57
(people who changed their minds based on new factual evidence)
Participation averages:
- Average answers/quote: 44
- Average answers/person: 1.81
- Average views of links/person: 1.76
- Average reviews/person: 0.79
Survey Responses
On Tuesday, Aug. 10th, we sent out a survey to all Truthsquad participants to seek their feedback. Early survey responses were generally favorable. In the first day after our survey went out, most survey respondents found the service useful or very useful -- and nearly half picked Truthsquad as their favorite service from NewsTrust this year.
Here are some sample quotes from these early responses:
- "Good oversight, with backup info on both sides. Liked the mix of liberal and conservative statements."
- "It gave a sense of mission - perhaps more so than regular NewsTrust reviewing."
- "I like that we - the people - are reviewing the statements versus 'journalists'."
- "Enjoyed having my cognitions tested. Checking stories takes a lot of time."
- "As far as politics go, I don't see how anyone can get at the truth other than to do their own factual research."
- "I was hoping that people would contribute and comment on matters they are either experts in or have direct knowledge of."
- "Not interested in being a journalist, editor, or fact checker. I just want a service that helps me find good journalism. Rating an article after I've read it is quick and low-cost to me. But fact checking is too much like real work."
- "The editors' section was quick, well-written and to the point. I also liked that participants were also checking ... the editors."
- "There are some people with the time and enthusiasm to undertake the work, and they should be compensated for that effort. I am not one of them, and have no problem contributing a few $$/month to fund that activity."
- "I'm totally won over by TruthSquad - I think it has a LOT of potential."
Thanks to our partners and advisors
We're deeply grateful to our amazing partners at the Poynter Institute and advisor at FactCheck.org for their invaluable contributions to this project.
In particular, we would like to thank Kelly McBride, Steve Myers and Julie Moos at Poynter Institute -- as well as Brooks Jackson at FactCheck.org. Heartfelt thanks to you all for making this project possible!
We asked our partners and advisors to share their general observations on why they participated in this pilot, what they learned from this experiment, what they found most and least useful about this approach -- and any other insights about this project.
Here are FactCheck.org director Brooks Jackson's personal comments about Truthsquad:
"FactCheck.org participated in this project because ordinary citizens need more help than they're getting from traditional news media to sort through the vast amount of disinformation to which they are exposed via the Internet and 24-hour cable talk networks. So far I would say the experiment is off to a solid start. The "verdicts" of the Truthsquad editors seem to me to be reasonable and based on good research. I'll be interested to learn how much of the research is the work of the "crowd" and how much comes from Truthsquad's paid staff.
One important observation. Even this brief initial experience has produced some pretty dramatic evidence of the limitations of crowd sourcing. The experience checking the Washington Times statement regarding a CitiGroup bonus shows that amateurs searching the Internet sometimes can't match the results obtained by an experienced staff reporter using old-fashioned shoe leather. In that case, one call to the reporter (or to Lew) might have produced the documentation needed to prove the accuracy of the claim."
Kelly McBride offered these thoughts as one of our editors on Truthsquad:
"It seems based on this pilot that citizens can learn fact-checking skills quite easily. The challenge is to motivate them to do this occasionally.
Some people love to do this stuff, but most are simply too busy. Adding more social gaming elements so folks accumulate points and prizes and compete against their friends - that might inspire some people."
Read Kelly's own report on Poynter:
Truthsquad Shows We Can 'Crowdfight' Culture of Misinformation
Steve Myers also shared his reflections on being a Truthsquad editor:
"The Truthsquad showed the opportunities and challenges of crowdsourcing. It's possible to fact-check many of the statements that are made every day in the media. But will people go to the effort to dig it up? Some will; some won't.
Many of the users seemed to respond based on their preconceptions -- who said it, how the statement aligned with their political views and what news outlet published it. Some people were upfront about this; they said because Bill O'Reilly said this or the Washington Times reported that, that is why they ruled the way they did.
And yet some people did take an interest and dug in to find the material online. When I ran into a roadblock in researching something, someone else picked up where I left off and found the correct document, read it, and drew a conclusion that I ended up agreeing with.
In writing a couple of verdicts, I was acutely aware of the judgment calls I was making -- what was the underlying fact that had to be proven or disproven, for instance. The most important judgment call: how much weight should I assign to something that was perhaps grammatically or technically correct but misleading? These quotes proved the cliche that there's a grain of truth in every untruth."
Thanks to our funders
This project was made possible in part by a generous grant from Omidyar Network, which funded a six-month investigation of new ways to help people separate fact from fiction online. We're very grateful for their support, as well as to the MacArthur Foundation and our other funding partners at Ashoka and the Ayrshire Foundation.
The Ford Foundation also contributed indirectly to this project, through its support of Poynter's Sense-Making Project, which led to the participation of Kelly McBride and Steve Meyers.
Thanks to our participants and team
Our deepest thanks go to all our participants in this Truthsquad, as well as thousands of new visitors -- and hundreds of new members who signed up during our pilot. These new participants came from across party lines and seemed to integrate well with the NewsTrust community.
Many thanks as well to our contributing editors Joey Baker, Kaizar Campwala, Kristin Gorski and Derek Hawkins, Jim Lang and many more for their inspiring participation in this project.
Last but not least, we are very grateful to our worldclass web development team, for their wonderful work in building Truthsquad: Subramanya Sastry, Caleb Waldorf, David Fox, created another great application of the NewsTrust platform in just a month, with the support of our staff editors Jon Mitchell and Kaizar Campwala. We're very lucky to have such a great team!
Next steps
This first pilot has given us a sense of what's possible -- and introduced our community to the discipline of factual verification, under the guidance of professionals from Poynter and Factcheck. Going forward, we will be posting new quotes to fact-check every Wednesday for the next few months (suggestions welcome: email us at truthsquad-at-newstrust-dot-net). But we can do better. Based on the positive response to this pilot, we would like to offer the Truthsquad service on an ongoing basis (we own the truthsquad.com domain).
To that end, we are looking for additional funds to provide Truthsquad on a daily basis this fall, through the mid-term elections. If you would like to support this project, please consider a donation to NewsTrust. Your contribution will help fight the rising tide of spin and misinformation on the Internet, as well as train citizens to evaluate the credibility of the news and information they find online.
We hope that Truthsquad can provide a useful fact-checking solution to the public, as well as help us all become more discerning about the news we consume.
Thanks for checking out Truthsquad!
-- by Fabrice Florin and Jon Mitchell, for the NewsTrust Team
Photo Credits: Fox News, News One, Associated Press, Good and Getty Images
