GEAR 2014 - Workshop on Gathering Efficient Assessments of Relevance

Evaluation is a fundamental part of Information Retrieval, and in the conventional Cranfield evaluation paradigm, sets of relevance assessments are a fundamental part of test collections. In this workshop, we wish to revisit how relevance assessments can be efficiently created. Potential themes include methods for generating assessments, the process of assessment, effort involved in assessing different materials, exploration of the concept of relevance etc. A discussion and exploration of this issue will be facilitated through the presentation of results based papers and position papers on the topic, as well as a group design activity.
Martin Halvey BSc (UCD, Ireland) PhD (UCD, Ireland) is a Lecturer (Assistant Professor) in the School of Engineering and Built Environment at Glasgow Caledonian University. His research focuses on user-centric issues in IR and novel interaction. He has a number of publications at high quality venues including ACM CHI, ACM SIGIR, ECIR, ACM Multimedia and IPM. Of these papers 3 of them have received best paper nominations. He has previously organized workshops at ACM Multimedia and for the Scottish Informatics and Computer Science Alliance (SICSA). He is the local organizing chair for ACM ICMR 2014 and one of the Lab Chairs for CLEF 2014. He is the current holder of a grant from the UK Arts and Humanities Research Council (AHRC) entitled "Understanding the annotation process: annotation for big data" (AH/L010364/1).
Robert Villa BSc (Strathclyde), MSc (Heriot-Watt), PhD (Glasgow) is a Lecturer (Assistant Professor) in the Information School (formerly the Department of Information Studies) in the faculty of Social Sciences University of Sheffield, UK. He is the Local Organiser for CLEF 2014. His research interests are primarily in the development and evaluation of search interfaces for text, image and video retrieval. Most of my past research has involved the creation and evaluation of novel search interfaces, including supporting exploratory search, collaborative search, and supporting artists and animators using content-based image and video retrieval. He is the current holder of a grant from the UK Arts and Humanities Research Council (AHRC) entitled "Understanding the annotation process: annotation for big data" (AH/L010364/1).
Paul D. Clough, BEng (York) PhD (Sheffield) PGCertHE (Sheffield) is a Senior Lecturer in the Information School (formerly the Department of Information Studies) in the faculty of Social Sciences University of Sheffield, UK. He is head of the Information Retrieval (IR) Group. He is currently Scientific Director of an EU-funded project called PATHS (Personalised Access To cultural Heritage Spaces), running an AHRC-funded project on recommender systems for WorldCat.org with OCLC Inc. and I am co-organiser of the TREC 2013 Session Track. He has also recently been awarded a Google Faculty Research Award for a project about developing a taxonomy of search sessions. His research interests include: information storage and retrieval, particularly multilingual searching of texts and images; evaluation of retrieval systems; natural language processing, text reuse and plagiarism detection.