[This entry was originally posted on the PBS MediaShift blog]
In June, Senator U.S. Senator Orrin Hatch made the statement that "87 million Americans will be forced out of their coverage" by President Obama's health care plan.
It was quite a claim. But was it true?
That's a common, and important, question -- and it can often be hard to quickly nail down the real facts in the information-overloaded world we live in. Professional fact-checking organizations like PolitiFact and FactCheck.org have taken up the charge to verify the claims of politicians, pundits and newsmakers, and they provide a great service to the public. But I believe there's also a role for the average person in the fact-checking process. By actively researching and verifying what we hear in the news, we can become more informed citizens, and more discriminating news consumers. These are essential skills for all of us to develop.
With that in mind, we at NewsTrust, a non-profit social news network, have been working on Truthsquad, a community fact-checking service that helps people verify the news online, with professional guidance.
Our first pilot for Truthsquad took place in August 2010, with the help of our partners at the Poynter Institute, our advisors at FactCheck.org and our funders at Omidyar Network. That pilot was well received by our community, partners and advisors, as noted in our first report, and by third-party observers such as GigaOm. We've since hosted a variety of weekly Truthsquads, and are starting a second pilot with MediaBugs.org and RegretTheError.com to identify and correct errors in the news media. (Disclosure: MediaShift managing editor Craig Silverman runs RegretTheError.com.)
Our first test project was by our standards a success; more importantly, it revealed several important lessons about the best ways to manage crowdsourced fact-checking, and about why people participate in this activity. Here are our key takeaways from this first pilot, which I'll elaborate on below:
- A game-like experience makes fact-checking more engaging.
- A professional-amateur (pro-am) collaboration delivers reliable results and a civil conversation.
- Crowd contributions are limited, requiring editorial oversight and better rewards.
- Community fact-checking fills a gap between traditional journalism and social media.
What is Truthsquad?
Truthsquad.com features controversial quotes from politicians or pundits and asks the community whether they think they are true or false. Community members are welcome to make a first guess, then check our answers and research links to see if they are correct. They can change their answer anytime, as they come across new facts.
To help participants find the right answer, we invite them to review and/or post links to factual evidence that supports or opposes each statement. A professional journalist leads each "truthsquad" to guide participants in this interactive quest. This "squad leader" then writes an in-depth verdict based on our collaborative research. That verdict is emailed to all participants, with request for comments. (It can be revised as needed.)
Finding #1: Game-Like Experience Makes Fact-Checking Engaging
We noted a significant increase in community participation for Truthsquad compared to other NewsTrust services we tested this year. Some data from our first pilot:
- This pilot generated twice as much participation as other 2010 pilots.
- Users gave ten times more answers per quote than reviews per story on our site.
- Over half of the participants read linked stories, and a third answered a Truthsquad quote.
- One in six participants reviewed the stories linked as factual evidence.
We think this high level of engagement is partly due to the game-like quality of our user experience, which starts by inviting people to guess whether a statement is true or false -- an easy task that anyone can do in under a minute.
After their first guess, people are more likely to participate as researchers, because their curiosity has been piqued and they want to know the answer. As a result, participants often take the time to review linked stories and post more evidence on their own. Without realizing it, they are becoming fact-checkers.
Finding #2: Pro-Am Collaboration Delivers Reliable Results
We decided early on that professionals needed to guide this collaborative investigation. We wanted to avoid some of the pitfalls of pure crowdsourcing initiatives, which can turn into mob scenes -- particularly around politically charged issues. At the start of this experiment, we asked experienced journalists at FactCheck.org and the Poynter Institute to coach us and our community and help write and edit some of our first verdicts.
We think the pro-am approach paid off in a number of ways:
- Amateurs learned valuable fact-checking skills by interacting with professionals.
- A few community members posted links that were critical to reaching our verdicts.
- Answers from our community generally matched final verdicts from our editors.
- We came to the same conclusions as FactCheck.org in side-by-side blind tests.
- Comments from participants were generally civil and focused on facts.
The results of our first pilot led our advisor Brooks Jackson, director at FactCheck.org, to comment, "So far I would say the experiment is off to a solid start. The verdicts of the Truthsquad editors seem to me to be reasonable and based on good research."
This collaboration between journalists and citizens made us all more productive. The professionals shared helpful information-gathering tips, and the citizens extended that expertise on a larger scale, with multiple checks and balances between our community and our editors. Our editors spearheaded this investigation, but the community made important contributions through comments and links to factual evidence (some of which were invaluable). On a couple occasions, we even revised our verdicts based on new evidence from our community. This focus on facts also helped set the tone for our conversations, which were generally civil and informative.
Finding #3: Crowd Contributions Are Limited, Requiring Better Rewards
Despite high levels of participation, we didn't get as many useful links and reviews from our community as we had hoped. Our editorial team did much of the hard work to research factual evidence. (Two-thirds of story reviews and most links were posted by our staff.) Each quote represented up to two days of work from our editors, from start to finish. So this project turned out to be more labor-intensive than we thought, and a daily fact-checking service will require a dedicated editorial team to guarantee reliable results.
Managing our community and responding thoughtfully to their posts also takes additional time, and is an important part of this process. In future releases, we would like to provide more coaching and educational services, as well as better rewards for our contributors.
Training citizens to separate fact from spin is perhaps the greatest benefit of our initiative, but keeping them engaged will require ingenuity and tender loving care on our part.
"It seems based on this pilot that citizens can learn fact-checking skills quite easily," said Kelly McBride of the Poynter Institute. "The challenge is to motivate them to do this occasionally."
To address this issue, future versions of Truthsquad could reward members who take the time to fact-check the news in order to get them to do it more often. We would like to give them extra points for reading, reviewing or posting stories, as well as special badges, redeemable credits and/or prizes. We can also feature high scores on leaderboards, and give monthly awards to the most deserving contributors.
Finding #4: Community Fact-Checking Fills a Need
Every survey we have done in recent years has consistently shown fact-checking as a top priority for our community, and this was confirmed by the results of this pilot.
Here are some key observations from our recent survey about NewsTrust's 2010 pilots:
- A majority of survey respondents (61 percent) found Truthsquad useful or very useful.
- Half of survey respondents wanted a new quote every day -- or several quotes a day.
- Half of survey respondents said they could fact-check quotes several times per week.
- One in seven survey respondents were prepared to donate for this service.
We think the generally favorable response to Truthsquad is due to two factors: a growing demand for fact-checking services, combined with a desire to contribute to this civic process. Fact-checking is still the best way to verify the accuracy of what people hear in the news, and it is perceived as an effective remedy to expose politicians or pundits who propagate misinformation.
At the same time, the explosion of social media makes people more likely to participate in investigations like these. They want this civil watchdog network, and expect to have a voice in it.
Next steps
Based on the lessons from this experiment, we would like to offer Truthsquad on an ongoing basis, with a goal to fact-check one quote a day, year-round -- as well as to feature the work of other trusted research organizations on Truthsquad.com.
We also want to let members post their own quotes for fact-checking and reward them for their contributions through both a game-like interface and more educational benefits. We have an opportunity to track the expertise of participants based on their answers, which could allow us to measure their progress with core news literacy skills, as well as their overall understanding of important public issues and the overall impact of our service.
Over time, we hope to provide more training and certification services, to build lasting research skills that could help students and adults alike play a more active role in our growing information economy. If this appeals to you, we invite you to sign up here and join our experiment.
As for that Orrin Hatch quote? In the end, 163 participants helped us fact-check his statement about health care. Our final verdict was that the Senator's claim was false. That finding was based on factual evidence provided by one of our NewsTrust members, who dug up the right set of regulations, and pointed out they had been misstated by Hatch. Our editor's verdict was confirmed by similar findings from FactCheck.org, and also matched our community's consensus: 138 participants answered that this statement was false (versus 11 who thought it was true).
More importantly, we as a community learned how to separate fact from fiction -- and became more engaged as citizens.
-- by Fabrice Florin
Comments