{"id":1528,"date":"2013-11-19T09:00:41","date_gmt":"2013-11-19T09:00:41","guid":{"rendered":"http:\/\/blogs.oii.ox.ac.uk\/policy\/?p=1528"},"modified":"2020-12-07T14:25:40","modified_gmt":"2020-12-07T14:25:40","slug":"verification-of-crowd-sourced-information-is-this-crowd-wisdom-or-machine-wisdom","status":"publish","type":"post","link":"https:\/\/ensr.oii.ox.ac.uk\/verification-of-crowd-sourced-information-is-this-crowd-wisdom-or-machine-wisdom\/","title":{"rendered":"Verification of crowd-sourced information: is this \u2018crowd wisdom\u2019 or machine wisdom?"},"content":{"rendered":"
\"Crisis
\u2018Code\u2019 or \u2018law\u2019? Image from an Ushahidi development meetup by afropicmusing<\/a>.<\/figcaption><\/figure>\n

In \u2018Code and Other Laws of Cyberspace<\/a>\u2019, Lawrence Lessig (2006) writes that computer code (or what he calls \u2018West Coast code\u2019) can have the same regulatory effect as the laws and legal code developed in Washington D.C., so-called \u2018East Coast code\u2019. Computer code impacts on a person\u2019s behaviour by virtue of its essentially restrictive architecture: on some websites you must enter a password before you gain access, in other places you can enter unidentified. The problem with computer code, Lessig argues, is that it is invisible, and that it makes it easy to regulate people\u2019s behaviour directly and often without recourse.<\/p>\n

For example, fair use provisions in US copyright law enable certain uses of copyrighted works, such as copying for research or teaching purposes. However the architecture of many online publishing systems heavily regulates what one can do with an e-book: how many times it can be transferred to another device, how many times it can be printed, whether it can be moved to a different format \u2013 activities that have been unregulated until now, or that are enabled by the law but effectively \u2018closed off\u2019 by code. In this case code works to reshape behaviour, upsetting the balance between the rights of copyright holders and the rights of the public to access works to support values like education and innovation.<\/p>\n

Working as an ethnographic researcher for Ushahidi<\/a>, the non-profit technology company that makes tools for people to crowdsource crisis information, has made me acutely aware of the many ways in which \u2018code\u2019 can become \u2018law\u2019. During my time at Ushahidi, I studied the practices that people were using to verify reports by people affected by a variety of events \u2013 from earthquakes to elections, from floods to bomb blasts. I then compared these processes with those followed by Wikipedians when editing articles about breaking news events. In order to understand how to best design architecture to enable particular behaviour, it becomes important to understand how such behaviour actually occurs in practice.<\/p>\n

In addition to the impact of code on the behaviour of users, norms, the market and laws also play a role. By interviewing both the users and designers of crowdsourcing tools I soon realized that \u2018human\u2019 verification, a process of checking whether a particular report meets a group\u2019s truth standards, is an acutely social process. It involves negotiation between different narratives of what happened and why; identifying the sources of information and assessing their reputation among groups who are considered important users of that information; and identifying gatekeeping and fact checking processes where the source is a group or institution, amongst other factors.<\/p>\n

One disjuncture between verification \u2018practice\u2019 and the architecture of the verification code developed by Ushahidi for users was that verification categories were set as a default feature, whereas some users of the platform wanted the verification process to be invisible to external users. Items would show up as being \u2018unverified\u2019 unless they had been explicitly marked as \u2018verified\u2019, thus confusing users about whether the item was unverified because the team hadn\u2019t yet verified it, or whether it was unverified because it had been found to be inaccurate. Some user groups wanted to be able to turn off such features when they could not take responsibility for data verification. In the case of the Christchurch Recovery Map<\/a> in the aftermath of the 2011 New Zealand earthquake, the government officials with whom volunteers who set up the Ushahidi instance were working wanted to be able to turn off such features because they were concerned that they could not ensure that reports were indeed verified and having the category show up (as \u2018unverified\u2019 until \u2018verified\u2019) implied that they were engaged in some kind of verification process.<\/p>\n

The existence of a default verification category impacted on the Christchurch Recovery Map group\u2019s ability to gain support from multiple stakeholders, including the government, but this feature of the platform\u2019s architecture did not have the same effect in other places and at other times. For other users like the original Ushahidi Kenya team who worked to collate instances of violence after the Kenyan elections in 2007\/08<\/a>, this detailed verification workflow was essential to counter the misinformation and rumour that dogged those events. As Ushahidi\u2019s use cases have diversified \u2013 from reporting death and damage during natural disasters to political events including elections, civil war and revolutions, the architecture of Ushahidi\u2019s code base has needed to expand. Ushahidi has recognised that code plays a defining role in the experience of verification practices, but also that code\u2019s impact will not be the same at all times, and in all circumstances. This is why it invested in research about user diversity in a bid to understand the contexts in which code runs, and how these contexts result in a variety of different impacts.<\/p>\n

A key question being asked in the design of future verification mechanisms is the extent to which verification work should be done by humans or non-humans (machines). Here, verification is not a binary categorisation, but rather there is a spectrum between human and non-human verification work, and indeed, projects like Ushahidi, Wikipedia and Galaxy Zoo<\/a> have all developed different verification mechanisms. Wikipedia uses a set of policies and practices about how content should be added and reviewed, such as the use of \u2018citation needed\u2019 tags for information that sounds controversial and that should be backed up by a reliable source. Galaxy Zoo uses an algorithm to detect whether certain contributions are accurate by comparing them to the same work by other volunteers.<\/p>\n

Ushahidi leaves it up to individual deployers of their tools and platform to make decisions about verification policies and practices, and is going to be designing new defaults to accommodate this variety of use. In parallel, Veri.ly<\/a>, a project by ex-Ushahidi Patrick Meier with organisations Masdar and QCRI is responding to the large amounts of unverified and often contradictory information that appears on social media following natural disasters by enabling social media users to collectively evaluate the credibility of rapidly crowdsourced evidence. The project was inspired by MIT\u2019s winning entry to DARPA’s \u2018Red Balloon Challenge\u2019<\/a> which was intended to highlight social networking\u2019s potential to solve widely distributed, time-sensitive problems, in this case by correctly identifying the GPS coordinates of 10 balloons suspended at fixed, undisclosed locations across the US. The winning MIT team<\/a> crowdsourced the problem by using a monetary incentive structure, promising $2,000 to the first person who submitted the correct coordinates for a single balloon, $1,000 to the person who invited that person to the challenge; $500 to the person who invited the inviter, and so on. The system quickly took root, spawning geographically broad, dense branches of connections. After eight hours and 52 minutes, the MIT team identified the correct coordinates for all 10 balloons.<\/p>\n

Veri.ly aims to apply MIT\u2019s approach to the process of rapidly collecting and evaluating critical evidence during disasters: \u201cInstead of looking for weather balloons across an entire country in less than 9 hours, we hope Veri.ly will facilitate the crowdsourced collection of multimedia evidence for individual disasters in under 9 minutes.\u201d It is still unclear how (or whether) Verily will be able to reproduce the same incentive structure, but a bigger question lies around the scale and spread of social media in the majority of countries where humanitarian assistance is needed. The majority of Ushahidi or Crowdmap<\/a> installations are, for example, still \u201csmall data\u201d projects, with many focused on areas that still require offline verification procedures (such as calling volunteers or paid staff who are stationed across a country, as was the case in Sudan<\/a> [3]). In these cases \u2013 where the social media presence may be insignificant — a team\u2019s ability to achieve a strong local presence will define the quality of verification practices, and consequently the level of trust accorded to their project.<\/p>\n

If code is law and if other aspects in addition to code determine how we can act in the world, it is important to understand the context in which code is deployed. Verification is a practice that determines how we can trust information coming from a variety of sources. Only by illuminating such practices and the variety of impacts that code can have in different environments can we begin to understand how code regulates our actions in crowdsourcing environments.<\/p>\n

For more on Ushahidi verification practices and the management of sources on Wikipedia during breaking news events, see:<\/p>\n

[1] Ford, H. (2012) Wikipedia Sources: Managing Sources in Rapidly Evolving Global News Articles on the English Wikipedia<\/a>. SSRN Electronic Journal. doi:10.2139\/ssrn.2127204<\/p>\n

[2] Ford, H. (2012) Crowd Wisdom<\/a>. Index on Censorship 41(4), 33\u201339. doi:10.1177\/0306422012465800<\/p>\n

[3] Ford, H. (2011) Verifying information from the crowd<\/a>. Ushahidi.<\/p>\n

\n

Heather Ford<\/a> has worked as a researcher, activist, journalist, educator and strategist in the fields of online collaboration, intellectual property reform, information privacy and open source software in South Africa, the United Kingdom and the United States. She is currently a DPhil student at the OII, where she is studying how Wikipedia editors write history as it happens in a format that is unprecedented in the history of encyclopedias. Before this, she worked as an ethnographer for Ushahidi. Read Heather’s blog<\/a>.<\/p>\n

For more on the ChristChurch Earthquake, and the role of digital humanities in preserving the digital record of its impact see: Preserving the digital record of major natural disasters: the CEISMIC Canterbury Earthquakes Digital Archive project<\/a> on this blog.<\/p>\n","protected":false},"excerpt":{"rendered":"

In \u2018Code and Other Laws of Cyberspace\u2019, Lawrence Lessig (2006) writes that computer code (or what he calls \u2018West Coast code\u2019) can have the same regulatory effect as the laws and legal code developed in Washington D.C., so-called \u2018East Coast code\u2019. Computer code impacts on a person\u2019s behaviour by virtue of its essentially restrictive architecture: […]<\/p>\n","protected":false},"author":83,"featured_media":1531,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[61,71,73,168,175,227],"_links":{"self":[{"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/posts\/1528"}],"collection":[{"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/users\/83"}],"replies":[{"embeddable":true,"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/comments?post=1528"}],"version-history":[{"count":1,"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/posts\/1528\/revisions"}],"predecessor-version":[{"id":3759,"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/posts\/1528\/revisions\/3759"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/media\/1531"}],"wp:attachment":[{"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/media?parent=1528"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/categories?post=1528"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ensr.oii.ox.ac.uk\/wp-json\/wp\/v2\/tags?post=1528"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}