SAD 2019 - 19 workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing (SAD 2019)

Ambiguity creates uncertainty in practically every facet of the web. Machine learning, recommender systems, web search, as well as news and systems that support human discourse, experience aspects of ambiguity and subjectivity that leads to inaccuracies, poor and undesirable performance. In crowdsourcing, human computation, and human-in-the-loop systems, topics central to TheWebConf, recent research has attempted to delve deeper into subjectivity and ambiguity, finding it to manifest in human-collected data primarily as disagreement [Cheatham and Hitzler, 2014; Plank, Hovy and Sogaard, 2014; Bayerl and Paul, 2011; Aroyo and Welty, 2015; Schaekermann, Law, Williams and Callaghan, 2016; Chang, Amershi and Kamar, 2017; Lin and Weld, 2014; etc.]. This includes the information presented to workers as part of a crowdsourcing task, the instructions for what to do with it, and the information they are asked to provide. These ambiguities become deeply tied into our machine learning models and metrics, as they are in the gold data. Similar ambiguity is found in interpreting and deriving utility from user generated data from large scale systems such as social media and search engines. Another aspect of disagreement surfaces in collaborative projects such as Wikipedia, online forums, and semantic markups. In language, ambiguity can result from missing details, contradictions and subjectivity. Subjectivity may stem from differences in cultural context, life experiences, or individual perception of hard-to-quantify properties. All of these can leave people with conflicting interpretations, leading to results that system builders would regard as "wrong". These issues share common ground with many areas of interest to TheWebConf.
SAD2019 (Subjectivity, Ambiguity and Disagreement in Crowdsourcing) Workshop aims to bring together a latent community of researchers who treat disagreement (and subjectivity and ambiguity) as signal, rather than noise. Such researchers use theoretical and empirical methodology to characterize, utilize, mitigate and derive value from subjectivity, ambiguity and disagreement. The workshop will include invited talks, short technical talks and a discussion of medium- and long-term challenges to fuel future work.
We encourage interdisciplinary submissions from the broad spectrum of crowdsourcing and human computation research in fields and application areas, such as computer science, information sciences, law, medical data analysis, communication science and political science, as well as those primarily working on human computation and crowdsourcing. Solutions to these challenging problems will benefit from a diverse set of perspectives. Topics of interest (but not limited to) are:
Interaction/relation between disagreement, ambiguity and subjectivity
Costs and challenges introduced by ambiguity
Designing tasks with high subjectivity and low inter-rater reliability (e.g., semantic, linguistic, common sense, moral judgements.)
Better metrics for characterizing disagreement (over traditional inter-rater reliability)
Ambiguity in human computation task design, how to identify it and what to do about it
Theoretical ambiguity-aware frameworks for collecting data
Teasing apart different sources of ambiguity
Best practices for collecting subjective data
Benchmarks and datasets for studying ambiguity
Grand challenges that will further our understanding of subjectivity, ambiguity and disagreement
Disagreement, ambiguity, subjectivity and their representation and interaction in different content modalities, e.g. text, images, videos, audio
Interdisciplinary perspectives on disagreement, ambiguity and subjectivity, e.g., law, political science, humanities
>>>>>> SUBMISSION >>>>>>
Authors can submit four types of papers:
short papers (up to 6 pages in length), plus unlimited pages for references
full papers (up to 10 pages in length), plus unlimited pages for references
position papers (up to 4 pages in length), plus unlimited pages for references
demo papers (up to 4 pages in length), plus unlimited pages for references
Submissions may present ideas, results from experimental studies, methodologies, work-in-progress and/or applications of systems that explore the topics of subjectivity, ambiguity and disagreement in crowdsourcing and human computation in relation to computational systems, applications, or services. Page limits include diagrams and appendices. Submissions should be formatted according to the formatting instructions in the General Guidelines.The proceedings of the workshop will be published jointly with the conference proceedings of TheWebConf 2019.
Submit papers through https://easychair.org/conferences/?conf=sad2019
All submissions must be written in English.