UIS 2015 - WORKSHOP: USEFULNESS OF INTERACTIVE IR SYSTEMS

Information Retrieval (IR) systems aim at helping a user to solve an information seeking problem. There is a long-standing tradition in IR research to measure the quality of an IR system in terms of the relevance of the documents returned (so-called Cranfield paradigm). However, due to the high interactivity of current IR systems it turned out that pure relevance-based measures fail to capture many factors that should be taken into account for IR evaluations, such as learning, task completion and effort. Current IR research therefore started to “take a broader perspective of the information seeking process to explicitly include users, tasks, and contexts in a dynamic setting” (Cole et al. 2009). This novel paradigm in IR evaluation expands the perspective to the entire search process instead of just evaluating single search results (cp. Dumais 2012). In addition, modern digital libraries offer more functionality besides search e.g. ? browsing relations, recommender, storing and structuring information, sharing information ? broadening the user-system interaction possibilities. These interactions lead to more valuable data for a better understanding of user needs and contexts and what is useful or not.
The notion of usefulness was first introduced by Cole et al. (2009) as a general criterion evaluating “how well the user is able to achieve their goal”. However, there is still a lack of computational usefulness metrics that can be taken to evaluate interactive IR systems.
The main goal of the workshop is to provide an international forum for discussing novel approaches that might contribute to an approximation of usefulness in interactive information retrieval. The workshop aims at bringing together experts from both ‘user’ and ‘system’ oriented information retrieval for a fruitful exchange of ideas and discussion how to tackle the evaluation of interactive IR from the perspective of usefulness.
The long-term research goal is to develop and evaluate new approaches for measuring usefulness of interactive IR systems. More specifically, we address questions such as:
What is usefulness and how can it be measured?
How can logging tools and frameworks look like to better capture usefulness?
How can usefulness be evaluated?
What can usefulness contribute to the improvement of interactive IR systems?
Workshop Topics
Contributions are solicited on, but not limited to, following topics:
Evaluation of interactive information retrieval
Information seeking behavior
Task based user modelling, interaction and personalization
Logging frameworks for sessions and tasks
Analyzing user behavior