View on GitHub


A resource page for a Eurovis 2018 State of the Art Report (STAR) on the to use of crowdsourcing in information visualization for evaluation.

Information Visualization Evaluation Using Crowdsourcing

This website specifies the full list of papers included in our survey titled “Information Visualization Evaluation Using Crowdsourcing” on crowdsourcing for evaluation in the information visualization community.


Rita Borgo, Informatics Department, Kings’s College London, United Kingdom

Luana Micallef, Helsinki Institute for Information Technology HIIT, Aalto University, Finland

Benjamin Bach, School of Informatics, University of Edinburgh, United Kingdom

Fintan McGee, Environmental Informatics Unit, Luxembourg Institute of Science and Technology, Luxembourg

Bongshin Lee, Microsoft Research, USA

Study Reporting Template

An editable version of the suggested study reporting template is available here (MS word Format): here

An latex version of the suggested study reporting template is available here (.tex file): here

Survey Papers

The raw survey data, including the coding of the papers, is downloadable as a tab delimited text file here

index Title Authors Year
1 Exploring the impact of emotion on visual judgement “Harrison, L.; Chang, R.; Aidong Lu” 2012
2 Priming Locus of Control to affect performance “Ottley, A.; Crouser, R.J.; Ziemkiewicz, C.; Chang, R.” 2012
3 Graphical Tests for Power Comparison of Competing Designs “Hofmann, H.; Follett, L.; Majumder, M.; Cook, D.” 2012
4 Learning Layouts for Single-PageGraphic Designs “O’Donovan, P.; Agarwala, A.; Hertzmann, A.” 2012
5 Does an Eye Tracker Tell the Truth about Visualizations?: Findings while Investigating Visualizations for Decision Making “Sung-Hee Kim; Zhihua Dong; Hanjun Xian; Upatising, B.; Ji Soo Yi” 2012
6 Perceptual Guidelines for Creating Rectangular Treemaps “Kong, N.; Heer, J.; Agrawala, M.” 2012
7 Interactive querying of temporal data using a comic strip metaphor J Jin, P Szekely 2010
8 Human computation in visualization: Using purpose driven games for robust evaluation ofvisualization algorithms “N Ahmed; Z Zheng; K Mueller” 2012
9 How locus of control influences compatibility with visualization style “C Ziemkiewicz; RJ Crouser, AR Yauilla” 2011
10 Perception of Average Value in Multiclass Scatterplots “Gleicher, M.; Correll, M.; Nothelfer, C.; Franconeri, S.” 2013
11 Ranking Visualizations of Correlation Using Weber’s Law “Harrison, L.; Fumeng Yang; Franconeri, S.; Chang, R.” 2014
12 How Hierarchical Topics Evolve in Large Text Corpora “Weiwei Cui; Shixia Liu; Zhuofeng Wu; Hao Wei” 2014
13 Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error “Correll, M.; Gleicher, M.” 2014
14 The persuasive power of data visualization “AV Pandey; A Manivannan; O Nov” 2014
15 How to display group information on node-link diagrams: an evaluation “R Jianu; A Rusu; Y Hu; D Taggart” 2014
16 Four Experiments on the Perception of Bar Charts “J Talbot; V Setlur; A Anand” 2014
17 The Perception of Visual UncertaintyRepresentation by Non-Experts “S Tak; A Toet; J van Erp” 2014
18 PEARL: an interactive visual analytic tool for understanding personal emotion style derived from social media “J Zhao; L Gou; F Wang; M Zhou” 2014
19 Lightness Constancy in Surface Visualization “Szafir, D.A.; Sarikaya, A.; Gleicher, M.” 2015
20 Guidelines for Effective Usage of Text Highlighting Techniques “Strobelt, H.; Oelke, D.; Bum Chul Kwon; Schreck, T.; Pfister, H.” 2016
21 Suggested Interactivity: Seeking Perceived Affordances for InformationVisualization “Boy, J.; Eveillard, L.; Detienne, F.; Fekete, J.-D.” 2016
22 Improving Bayesian reasoning: the effects of phrasing, visualization, and spatial ability “A Ottley; EM Peck; LT Harrison;” 2016
23 Map LineUps: effects of spatial structure on graphical inference “Beecham, Roger; Dykes, Jason ;Meulemans, Wouter ; Slingsby, Aidan; Turkay, Cagatay ; Wood, Jo” 2016
24 Optimizing Hierarchical Visualizations with the Minimum Description Length Principle “Veras, Rafael ;Collins, Christopher” 2016
25 Evaluating the Impact of Binning 2D Scalar Fields “Lace Padilla; P. Samuel Quinan; Miriah Meyer; Sarah H. Creem-Regehr” 2016
26 HindSight: Encouraging Exploration through Direct Encoding of Personal Interaction History “Mi Feng; Cheng Deng; Evan M. Peck; Lane Harrison” 2016
27 A Study On Designing Effective Introductory Materials for Information Visualization “Yuzuru Tanahashi; Nick Leaf; Kwan-Liu Ma” 2016
28 An Evaluation of the Impact of Visual Embellishments in Bar Charts “Drew Skau; Lane Harrison; Robert Kosara” 2015
29 GraphUnit: Evaluating Interactive Graph Visualizations Using Crowdsourcing “Mershack Okoe; Radu Jianu” 2015
30 Interaction with uncertainty in visualisations “Advait Sarkar; Alan F Blackwell; Mateja Jamnik; Martin Spott” 2014
31 Preconceptions and Individual Differences in Understanding Visual Metaphors “Caroline Ziemkiewicz; Robert Kosara” 2009
32 Selecting Semantically-Resonant Colors for Data Visualization “Sharon Lin; Julie Fortuna; Chinmay Kulkarni; Maureen Stone; Jeffrey Heer” 2013
33 Arcs, Angles, or Areas: Individual Data Encodings in Pie and Donut Charts “Drew Skau;Robert Kosara” 2016
34 How Ordered Is It? On the Perceptual Orderability of Visual Channels “David H. S. Chung; Daniel Archambault; Rita Borgo; Darren J. Edwards; Robert S. Laramee; Min Chen” 2016
35 Judgment Error in Pie Chart Variations “Robert Kosara; Drew Skau” 2016
36 Using icicle trees to encode the hierarchical structure of source code “I. Bacher;B. Mac Namee; J. D. Kelleher” 2016
37 Evaluating Viewpoint Entropy for Ribbon Representation of Protein Structure “J. Heinrich; J. Vuong; C. J. Hammang; A. Wu; M. Rittenbruch; J. Hogan; M. Brereton; S. I. O’Donoghue” 2016
38 Mimic: visual analytics of online micro-interactions “Simon Breslav; Azam Khan; Kasper Hornbæk” 2014
39 Progressive Parallel Coordinates Rene Rosenbaum, Jian Zhi, Bernd Hamann 2012
40 Crowdsourcing graphical perception: using mechanical turk to assess visualization design “Heer, Jeffrey;Michael Bostock” 2010
41 The impact of social information on visual judgments “Hullman, Jessica; Eytan Adar; Priti Shah” 2011
42 Playable data: characterizing the design space of game-y infographics “Diakopoulos, Nicholas; Funda Kivran-Swaine; Mor Naaman” 2011
43 Strategies for crowdsourcing social data analysis “Wesley Willett?; Jeffrey Heer†; Maneesh Agrawala?” 2012
44 Comparing averages in time series data “Correll, M.; Albers, D.M; Franconeri, S.; Gleicher, M.” 2012
45 Influencing visual judgment through affective priming “Harrison, L.; Skau, D.; Franconeri, S.; Lu, A.; Chang, R” 2013
46 Modeling how people extract color themes from images “Sharon Lin; Pat Hanrahan” 2013
47 Extracting references between text and charts via crowdsourcing Nicholas Kong, Marti A. Hearst, Maneesh Agrawala 2014
48 Automatic generation of semantic icon encodings for visualizations Vidya Setlur, Jock D. Mackinlay 2014
49 Task-driven evaluation of aggregation in time series visualization Danielle Albers, Michael Correll, Michael Gleicher 2014
50 A table!: improving temporal navigation in soccer ranking tables Charles Perin, Romain Vuillemot, Jean-Daniel Fekete 2014
51 Understand users’ comprehension and preferences for composing information visualizations Huahai Yang, Yunyao Li, Michelle X. Zhou 2014
52 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman 2015
53 “Without the Clutter of Unimportant Words”: Descriptive Keyphrases for Text Visualization JASON CHUANG, CHRISTOPHER D. MANNING, and JEFFREY HEER, 2012
54 Infographic Aesthetics: Designing for the First Impression Lane Harrison, Katharina Reinecke, Remco Chang 2015
55 ISOTYPE Visualization: Working Memory, Performance, and Engagement with Pictographs Steve Haroz, Robert Kosara, and Steven L. Franconeri. 2015
56 How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques. Anshul Vikram Pandey, Katharina Rall, Margaret L. Satterthwaite, Oded Nov, and Enrico Bertini. 2015
57 A Comparative Evaluation on Online Learning Approaches using Parallel Coordinate Visualization Kwon, Bum Chul, and Bongshin Lee. 2016
58 The Effect of Visual Appearance on the Performance of Continuous Sliders and Visual Analogue Scales Justin Matejka, Michael Glueck, Tovi Grossman, and George Fitzmaurice 2016
59 When (ish) is My Bus? User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems Kay, Matthew, Tara Kola, Jessica R. Hullman, and Sean A. Munson 2016
60 A Deeper Understanding of Sequence in Narrative Visualization Hullman, J., Drucker, S., Riche, N. H., Lee, B., Fisher, D., & Adar, E. 2013
61 Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing Micallef, L., Dragicevic, P., Fekete, J. 2012
62 Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty Nadia Boukhelifa, Anastasia Bezerianos, Tobias Isenberg, Jean-Daniel Fekete 2012
63 HOLA: Human-like Orthogonal Network Layout. Visualization and Computer Graphics, Steve Kieffer, Tim Dwyer, Kim Marriott, Michael Wybrow. 2016
64 Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis “Willett, W.; Ginosar, S.; Steinitz, A.; Hartmann, B.; Agrawala, M.” 2013
65 Laws of Attraction: From Perceived Forces to Conceptual Similarity Caroline Ziemkiewicz and Robert Kosara 2010
66 Learning Perceptual Kernels for Visualization Design Demiralp, C. D., Bernstein, M. S., & Heer, J. 2014
67 A Crowdsourced Alternative to Eye-tracking for Visualization Understanding Nam Wook Kim, Aude Olivia, Zoya Bylinskii Krzysztof Z. Gajos, Michelle A. Borkin, Hanspeter Pfister 2015
68 Quantity estimation in visualizations of tagged text Correll, Michael A., Eric C. Alexander, and Michael Gleicher. 2013
69 Explaining the Gap: Visualizing One’s Predictions Improves Recall and Comprehension of Data Yea-Seul Kim, Katharina Reinecke, Jessica Hullman 2017
70 Regression by Eye: Estimating Trends in Bivariate Visualizations Michael Correll, Jeffrey Heer 2017
71 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. 2017
72 Assessing User Engagement in Information Visualization Ya-Hsin Hung,Paul Parsons 2017
73 Narratives in Crowdsourced Evaluation of Visualizations: A Double-Edged Sword? Evanthia Dimara, Anastasia Bezerianos,Pierre Dragicevic 2017
74 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister 2017
75 A Crowdsourced Approach to Colormap Assessment Terece L. Turton, Colin Ware, Francesca Samsel, David H. Rogers 2017
76 Finding a Clear Path: Structuring Strategies for Visualization Sequences Jessica Hullman, Robert Kosara, and Heidi Lam 2017
77 Visual Narrative Flow: Exploring Factors Shaping Data Visualization Story Reading Experiences S. McKenna, N. Henry Riche, B. Lee, J. Boy, & M. Meyer 2017
78 Readability and Precision in Pictorial Bar Charts Drew Skau and Robert Kosara 2017
79 Fauxvea: Crowdsourcing Gaze Location Estimates for Visualization Analysis Tasks S. R. Gomez, R. Jianu, R. Cabeen, H. Guo and D. H. Laidlaw 2016
80 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf 2017
81 Blinded with Science or Informed by Charts? A Replication Study Pierre Dragicevic, Yvonne Jansen 2017
82 Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries Cristian Felix, Steven Franconeri and Enrico Bertini 2017