Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study

Amber Shinsel; Todd Kulesza; Margaret Burnett; William Curran; Alex Groce; Simone Stumpf; Weng Keen Wong

doi:10.1109/VLHCC.2011.6070377

Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study

Amber Shinsel, Todd Kulesza, Margaret Burnett, William Curran, Alex Groce, Simone Stumpf, Weng Keen Wong

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

9 Scopus citations

Abstract

Intelligent assistants sometimes handle tasks too important to be trusted implicitly. End users can establish trust via systematic assessment, but such assessment is costly. This paper investigates whether, when, and how bringing a small crowd of end users to bear on the assessment of an intelligent assistant is useful from a cost/benefit perspective. Our results show that a mini-crowd of testers supplied many more benefits than the obvious decrease in workload, but these benefits did not scale linearly as mini-crowd size increased - there was a point of diminishing returns where the cost-benefit ratio became less attractive.

Original language	English (US)
Title of host publication	Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011
Pages	47-54
Number of pages	8
DOIs	https://doi.org/10.1109/VLHCC.2011.6070377
State	Published - 2011
Externally published	Yes
Event	2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011 - Pittsburgh, PA, United States Duration: Sep 18 2011 → Sep 22 2011

Publication series

Name	Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011

Conference

Conference	2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011
Country/Territory	United States
City	Pittsburgh, PA
Period	9/18/11 → 9/22/11

Keywords

crowdsourcing
end-user programming
machine learning
testing

ASJC Scopus subject areas

Computer Vision and Pattern Recognition
Human-Computer Interaction
Software

Access to Document

10.1109/VLHCC.2011.6070377

Cite this

Shinsel, A., Kulesza, T., Burnett, M., Curran, W., Groce, A., Stumpf, S., & Wong, W. K. (2011). Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study. In Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011 (pp. 47-54). Article 6070377 (Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011). https://doi.org/10.1109/VLHCC.2011.6070377

Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study. / Shinsel, Amber; Kulesza, Todd; Burnett, Margaret et al.
Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011. 2011. p. 47-54 6070377 (Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Shinsel, A, Kulesza, T, Burnett, M, Curran, W, Groce, A, Stumpf, S & Wong, WK 2011, Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study. in Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011., 6070377, Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011, pp. 47-54, 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011, Pittsburgh, PA, United States, 9/18/11. https://doi.org/10.1109/VLHCC.2011.6070377

Shinsel A, Kulesza T, Burnett M, Curran W, Groce A, Stumpf S et al. Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study. In Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011. 2011. p. 47-54. 6070377. (Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011). doi: 10.1109/VLHCC.2011.6070377

@inproceedings{57e6d7fffa9747bb97264c71d4ccf971,

title = "Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study",

abstract = "Intelligent assistants sometimes handle tasks too important to be trusted implicitly. End users can establish trust via systematic assessment, but such assessment is costly. This paper investigates whether, when, and how bringing a small crowd of end users to bear on the assessment of an intelligent assistant is useful from a cost/benefit perspective. Our results show that a mini-crowd of testers supplied many more benefits than the obvious decrease in workload, but these benefits did not scale linearly as mini-crowd size increased - there was a point of diminishing returns where the cost-benefit ratio became less attractive.",

keywords = "crowdsourcing, end-user programming, machine learning, testing",

author = "Amber Shinsel and Todd Kulesza and Margaret Burnett and William Curran and Alex Groce and Simone Stumpf and Wong, {Weng Keen}",

year = "2011",

doi = "10.1109/VLHCC.2011.6070377",

language = "English (US)",

isbn = "9781457712456",

series = "Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011",

pages = "47--54",

booktitle = "Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011",

note = "2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011 ; Conference date: 18-09-2011 Through 22-09-2011",

}

TY - GEN

T1 - Mini-crowdsourcing end-user assessment of intelligent assistants

T2 - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011

AU - Shinsel, Amber

AU - Kulesza, Todd

AU - Burnett, Margaret

AU - Curran, William

AU - Groce, Alex

AU - Stumpf, Simone

AU - Wong, Weng Keen

PY - 2011

Y1 - 2011

N2 - Intelligent assistants sometimes handle tasks too important to be trusted implicitly. End users can establish trust via systematic assessment, but such assessment is costly. This paper investigates whether, when, and how bringing a small crowd of end users to bear on the assessment of an intelligent assistant is useful from a cost/benefit perspective. Our results show that a mini-crowd of testers supplied many more benefits than the obvious decrease in workload, but these benefits did not scale linearly as mini-crowd size increased - there was a point of diminishing returns where the cost-benefit ratio became less attractive.

AB - Intelligent assistants sometimes handle tasks too important to be trusted implicitly. End users can establish trust via systematic assessment, but such assessment is costly. This paper investigates whether, when, and how bringing a small crowd of end users to bear on the assessment of an intelligent assistant is useful from a cost/benefit perspective. Our results show that a mini-crowd of testers supplied many more benefits than the obvious decrease in workload, but these benefits did not scale linearly as mini-crowd size increased - there was a point of diminishing returns where the cost-benefit ratio became less attractive.

KW - crowdsourcing

KW - end-user programming

KW - machine learning

KW - testing

UR - http://www.scopus.com/inward/record.url?scp=82855181173&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=82855181173&partnerID=8YFLogxK

U2 - 10.1109/VLHCC.2011.6070377

DO - 10.1109/VLHCC.2011.6070377

M3 - Conference contribution

AN - SCOPUS:82855181173

SN - 9781457712456

T3 - Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011

SP - 47

EP - 54

BT - Proceedings - 2011 IEEE Symposium on Visual Languages and Human Centric Computing, VL/HCC 2011

Y2 - 18 September 2011 through 22 September 2011

ER -

Mini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this