Home About Browse Search
Svenska


Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game

Salk, Carl and See, Linda and Sturn, Tobias and Fritz, Steffen and Perger, Christoph (2016). Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game. International Journal Of Digital Earth. 9:4, 410-426
[Journal article]

[img]
Preview
PDF - Accepted Version
653kB

Official URL: http://dx.doi.org/10.1080/17538947.2015.1039609

Abstract

Volunteered geographic information (VGI) is the assembly of spatial information based on public input. While VGI has proliferated in recent years, assessing the quality of volunteer-contributed data has proven challenging, leading some to question the efficiency of such programs. In this paper, we compare several quality metrics for individual volunteers' contributions. The data were the product of the Cropland Capture' game, in which several thousand volunteers assessed 165,000 images for the presence of cropland over the course of 6 months. We compared agreement between volunteer ratings and an image's majority classification with volunteer self-agreement on repeated images and expert evaluations. We also examined the impact of experience and learning on performance. Volunteer self-agreement was nearly always higher than agreement with majority classifications, and much greater than agreement with expert validations although these metrics were all positively correlated. Volunteer quality showed a broad trend toward improvement with experience, but the highest accuracies were achieved by a handful of moderately active contributors, not the most active volunteers. Our results emphasize the importance of a universal set of expert-validated tasks as a gold standard for evaluating VGI quality.

Authors/Creators:Salk, Carl and See, Linda and Sturn, Tobias and Fritz, Steffen and Perger, Christoph
Title:Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game
Series/Journal:International Journal Of Digital Earth (1753-8955)
Year of publishing :2016
Volume:9
Number:4
Page range:410-426
Number of Pages:17
Publisher:Taylor & Francis
ISSN:1753-8955
Language:English
Publication Type:Journal article
Refereed:Yes
Article category:Scientific peer reviewed
Version:Accepted version
Full Text Status:Public
Agris subject categories.:U Auxiliary disciplines > U30 Research methods
Subjects:(A) Swedish standard research categories 2011 > 1 Natural sciences > 105 Earth and Related Environmental Sciences > Environmental Sciences (social aspects to be 507)
(A) Swedish standard research categories 2011 > 1 Natural sciences > 105 Earth and Related Environmental Sciences > Geosciences, Multidisciplinary
Keywords:crowdsourcing, volunteered geographic information, cropland, data quality, image classification, Geo-Wiki
URN:NBN:urn:nbn:se:slu:epsilon-e-4139
Permanent URL:
http://urn.kb.se/resolve?urn=urn:nbn:se:slu:epsilon-e-4139
Additional ID:
Type of IDID
Web of Science (WoS)000374661000005
ID Code:14273
Faculty:S - Faculty of Forest Sciences
Department:(S) > Southern Swedish Forest Research Centre
Deposited By: SLUpub Connector
Deposited On:28 Apr 2017 07:50
Metadata Last Modified:28 Apr 2017 07:50

Repository Staff Only: item control page

Downloads

Downloads per year (since September 2012)

View more statistics

Downloads
Hits