EvalUMAP 2017 - Workshop on Towards comparative evaluation in user modeling, adaptation and personalization

Research in the areas of User Modelling, Adaptation and Personalization faces a number of significant scientific challenges. One of the most significant of these challenges is the issue of comparative evaluation. It has always been difficult to rigorously compare different approaches to personalization, as the function of the resulting systems is, by their nature, heavily influenced by the behaviour of the users involved in trialling the systems. To-date this topic has received relatively little attention. Developing comparative evaluations in this space would be a huge advancement as it would enable shared comparison across research, which to-date has been very limited.
Taking inspiration from communities, such as Information Retrieval and Machine Translation, the EvalUMAP Workshop series seek to propose and design one or more shared tasks to support the comparative evaluation of approaches to User Modelling, Adaptation and Personalization. This year’s workshop will solicit presentations from key practitioners in the field on innovative datasets that meet specific requirements (e.g. ownership, accessibility, privacy) and that could form the basis to start scoping and designing shared task-based challenges and evaluations in the area of user adaptation and personalisation for next year. The resulting shared task(s) will be accompanied by appropriate models, content, metadata, user behaviours, etc., and can be used to comprehensively compare how different approaches and systems perform. In addition, a number of evaluation metrics and methods will be outlined, that participants would be expected to perform in order to facilitate comparison. Finally, the proposed shared task(s) will be disseminated in the community and the resulting outcomes will be presented at an EvalUMAP forum next year.
In particular, the planned outcomes of the EvalUMAP Workshop 2017 is as follows: (1) A clear understanding of the challenges and requirements related to the design of a shared task-based approach in User Modeling, Adaptation and Personalization space and (2) identify specific issues and requirements on user data capturing and dataset processing in the context of personalization for a shared task (3) the identification and description of suitable and publicly accessible datasets that overcome the previous identified challenges and (4) the design of shared task-based evaluations using suitable datasets that will take place throughout 2017 and early 2018, and be presented at UMAP 2018.
Workshop topics are evaluation focused and include, but are not limited to:
Understanding UMAP evaluation
Defining tasks and scenarios for evaluation purposes
Identification of potential corpora (datasets) for shared tasks
Automated and semi automated processes on creating appropriate datasets and simulating user behaviours etc in order to accommodate a shared task
Interesting target tasks and explanations of their importance
Critiques or comparisons of existing evaluation metrics and methods
How we can combine existing evaluation metrics and methods?
Reusing or improving previously suggested metrics and methods
Reducing the cost of evaluation
Proposal of new evaluation metrics and methods
Technical challenges associated with design and implementation
Anonymization of datasets, Privacy, Ethics and Security issues on the use of datasets