CEPDC 2012 - International Workshop on Community IR Evaluation using Private Data Collections

Community IR Evaluation using Private Data Collections
Ian Soboroff (NIST), Paul Over(NIST), Ellen Voorhees (NIST), Ophir Frieder (Georgetown University)
Hide Description
Information retrieval experiment has thrived on the basis of common, shared test collection resources. From the earliest test collections to those produced by TREC, CLEF, NTCIR, INEX, and other evaluation workshops, the foundational principle is that test collections are a community resource, possibly built and definitely used by multiple organizations, for their own research goals as well as the broader goal of defining measurement methods for information access and search tasks. This principle perseveres because the most efficient and effective methods for test collection creation that we know of work best within a community model such as pooling.