LL 2014 - Workshop on Living Labs for Information Retrieval Evaluation

In the past few years, a new evaluation methodology known as living labs has been proposed as a way for researchers to be able to perform in-situ evaluation which involve and integrate users within the research process. The basic idea of living labs for information retrieval (IR) is that rather than individual research groups independently developing experimental search infrastructures and gathering their own groups of test searchers for IR evaluations, a central and shared experimental environment is developed to facilitate the sharing of resources in a live setting (most important of all: users).
Living labs would offer huge benefits to the community, such as: availability of, potentially larger, cohorts of real users and their behaviours, e.g. querying behaviours, for experiment purposes; cross-comparability across research centres; and greater knowledge transfer between industry and academia, when industry partners are involved. The need for this methodology is further amplified by the increased reliance of IR approaches on proprietary data; living labs are a way to bridge the data divide between academia and industry. Progress towards realising actual living labs has nevertheless been limited. There are many challenges to be overcome before the benefits associated with living labs for IR can be realised, including challenges associated with living labs architecture and design, hosting, maintenance, security, privacy, participant recruiting, and scenarios and tasks for use development.
The 1st Living Labs for Information Retrieval Evaluation (LL’13) workshop at CIKM 2013 was a first attempt to bring people, both from academia and industry, together to discuss challenges and to formulate practical next steps. The workshop was successful in identifying and documenting possible further directions. The goal of the second edition of the workshop is to continue our community building efforts around living labs for IR and to pursue the directions set out at LL’13. As part of this, a challenge with shared tasks in the e-commerce space and local domain search is planned.
Possible use cases in this space include, but are not limited to:
The e-commerce domain (i.e., product search and recommendation)
Local domain search (e.g., university search engines)
The personal search space (search of personal computer files, emails, web pages looked at, etc.)
Medical information retrieval (e.g., patients searching for medical information on the Internet)
Searching Wikipedia (an open-source collection with fewer privacy concerns than, say, personal search)
Workshop topics include, but are not limited to:
Privacy and security:
Hosting data on secure server
Gaining subjects trust
Coping with individuals need for privacy
Alternates when individuals will not share their data
Legal and ethical issues:
User consent
Ethics approval
Legalities regarding release of data
Trust between parties
Copyright issues
Commercial sensitivity of interaction data
Technical challenges:
Designing and implementing living labs architecture
Cost of implementation
Maintenance and adoption
Managing living labs infrastructure
Practical challenges:
Forming living labs for IR partners within the research community
Obtaining commercial partners
Alternates when commercial partners cannot be obtained
Defining tasks and scenarios for evaluation purposes