RepEval 2016 - First Workshop on Evaluating Vector Space Representations for NLP

Models that learn real-valued vector representations of words, phrases, sentences, and even documents are ubiquitous in today's NLP landscape. These representations are usually obtained by training a model on large amounts of unlabeled data, and then employed in NLP tasks and downstream applications. While such representations should ideally be evaluated according to their value in these applications, doing so is laborious, and it can be hard to rigorously isolate the effects of different representations for comparison.
The Challenge
There is therefore an important need for better ways to evaluate semantic representations of language via simple and generalizable proxy tasks. To date, these proxy tasks have been mainly focused on lexical similarity and relatedness, and do not capture the full spectrum of interesting linguistic properties that are useful for downstream applications. This workshop challenges its participants to propose methods and/or design benchmarks for evaluating the next generation of vector space representations, for presentation and detailed discussion at the event. Following the workshop, the highest-quality proposals will receive the support of the organizers and participants, and some financial support, to help produce their proposed resource to the highest standard.
Related Workshop
Prospective attendees of RepEval may be interested in the following companion workshop, also collocated with ACL: the 1st Workshop on Representation Learning for NLP, organised by Phil Blunsom, Kyunghyun Cho, Shay Cohen, Edward Grefenstette, Karl Moritz Hermann, Laura Rimmel, Jason Weston and Scott Yih. We expect the topics covered by both workshops to complement one another well, and encourage ACL attendees to consider attending both.