DESI 2011 - ICAIL 2011 Workshop on Setting Standards for Searching Electronically Stored Information in Discovery Proceedings (DESI IV Workshop)

ICAIL 2011 Workshop on
Setting Standards for
Searching Electronically Stored Information
in Discovery Proceedings
(DESI IV Workshop)

June 6, 2011, University of Pittsburgh, Pittsburgh, PA, USA

Purpose | Background | Organization of the Workshop | Participation | Submissions
Important Dates | DESI History | Standards | References | Organizing Committee
Purpose

The DESI IV workshop is intended to provide a platform for discussion of an open standard governing the elements of a state-of-the-art search for electronic evidence in the context of civil discovery. The dialog at the workshop might take several forms, ranging from a straightforward discussion of how to measure and improve upon the "quality" of existing search processes; to discussing the creation of a national or international recognized standard on what constitutes a "quality process" when undertaking e-discovery searches; to a more ambitious discussion concerning creation of a type of standards "authority" or certification entity that could certify compliance with whatever standard emerges. Issues to be considered include the potential benefits of a standard (e.g., reducing the need for evidentiary hearings on the reasonableness of a process), versus its potential costs (e.g., the risk of inhibiting innovation through early convergence on rigid standards), and timelines.
Background

The Sedona Conference®, a leading legal think tank, recognized in 2009 that "the legal profession is at a crossroads," in terms of its willingness to embrace automated, analytical and statistical approaches in the area of e-discovery. However, while in the last four years a cottage industry of published case law and commentaries have come into existence recognizing that parties in litigation should reach some kind of agreement on what constitutes a "search protocol," there is no widely agreed-upon set of standards or best practices for how to conduct a reasonable e-discovery search for relevant evidence. The Sedona Conference and others have, however, called out to industry and academia to assist in the further development of standards of what constitutes "best practices" in the area of performing searches of electronic evidence.
In initiating a discussion about standards for what constitutes a "quality" process in e-discovery search, the workshop will serve to achieve the aim of bringing together academia and industry in the development of standards in this area. A recent article on "Evaluation of Information Retrieval in E-Discovery" in the journal Artificial Intelligence and Law's special issue on E-Discovery suggested that:

"One broad class of approaches that has gained currency in recent years ... is known as 'process quality.' Essentially, ... the important thing is that we agree on how each performer of E-discovery services should design measures to gain insight into the quality of the results achieved by their particular process. The design of their process, and of their specific measures, is up to each performer. Of course, some performers might benefit from economies of scale by adopting measures designed by others, but because the measures must fit the process and because process innovation should be not just accommodated but encouraged, forced convergence on specific measures can be counterproductive. So process quality approaches seek to certify the way in which the measurement process is performed rather than what specifically is measured."

Organization of the Workshop

The full day workshop will be organized in four parts:
Part I: The workshop will begin with an overview of recent developments in the area of e-discovery search, including a presentation of recent case law, the needs of the legal profession in the area of reducing cost through better use of automated methods, and recent work on evaluation design.

Part II will consist of a set of brief presentations selected to illustrate the diversity of present e-discovery search processes (ranging from fully manual, to machine assisted, to fully automatic) and a second set of brief presentations to illustrate the diversity of present standard-setting efforts in the area of "quality" (e.g., ISO 9000 family of international quality management systems standards, and Capability Maturity Model Integration (CMMI) in software engineering).

Part III will consist of a set of breakout sessions to engage the workshop participants in brainstorming with respect to what process quality standards for e-discovery search might entail. Each breakout group will be asked to initially look at the problem from the perspective of some specific process that was introduced in Part II. This discussion will begin with informal discussions over lunch, and then it will be the principal focus of the first session after lunch, ending with brief reports from each group.

Part IV will be a facilitated session with panelists representing the various breakout groups discussing the ideas that have emerged, and further discussion with all participants. The panel will conclude by focusing on recommendations for further work, building from two questions:

What research questions should be explored, so as to contribute to the development of process quality standards in the e-discovery search area?
Who, beyond those already in the room, do we need to engage with to address the issues that we have identified?
Participation

We invite both e-discovery stakeholders and practitioners from the law, government, and industry, along with researchers on process quality, information retrieval, human language technology, human-computer interaction, artificial intelligence, and other fields connected with e-discovery.
Submissions

To help craft the program, we encourage the submission of research papers and position papers on emerging best practices in e-discovery search as well as papers discussing the efficacy of standards setting in this area. Accepted position papers and accepted research papers will be made available on the Workshop's Web page and distributed to participants on the day of the event, and some speakers may be selected from among those submitting position papers. See the Call for Submissions for submission details.
Important Dates

Research papers due: April 1, 2011
Position papers due: April 22, 2011
Notification for research papers: April 22, 2011
Preliminary Agenda posted: May 9, 2011
Workshop: June 6, 2011
It is anticipated that the Workshop will begin at approximately 9:00 AM and that it will end by 5:00 PM.
DESI History

At the beginning of the second decade of the 21st century, lawyers continue to increasingly face the problem of how to effectively and efficiently conduct searches for relevant documents across increasingly complex, enterprise-wide collections within corporate and institutional settings. Hundreds of millions of documents are now routinely subject to searches across a wide spectrum of litigation and investigatory contexts (e.g., the Lehman Brothers investigation necessitated a review of 350 million pages or 3.5 petabytes of material). In recognition of this situation, in 2006 the United States adopted new rules governing civil litigation in federal courts. These courts have recognized "electronically stored information" (ESI) as a term of art, embracing all forms of electronic documents made subject to the civil discovery process. Under the new rules, opposing parties in federal court litigation now have an early "meet and confer" duty to discuss a range of electronic discovery ("e-discovery") issues, including the continued storage, preservation, and access to ESI in their respective physical and legal custodies.
DESI IV follows three successful prior DESI (Discovery of Electronically Stored Information) Workshops: at ICAIL 2007 (DESI I, Palo Alto) and ICAIL 2009 (DESI III, Barcelona), and an intermediate workshop (DESI II) sponsored by University College London in 2008. In DESI I, a wide array of individuals came together for perhaps the first time to foster engagement between e-discovery practitioners and a broad range of research communities who can contribute to the development of new technologies to support the e-discovery process. The DESI II and III workshops broadened the scope of this discussion to include comparisons of requirements between different national settings and different legal contexts. DESI IV builds on these efforts with its emphasis on standards-setting or benchmarking for e-discovery searches.

Standards

There may well be many good ideas that we can draw on that have been worked out in the context of existing standards-setting processes in other fields, such as:
ISO 9000 family of international quality measurement system standards
Payment Card Industry Data Security Standard
Statement on Auditing Standards No. 70: Service Organizations
Capability Maturity Model Integration
Transaction Processing Performance Council's benchmarks
NIST Computer Forensics Tool Testing Program
Department of Defense Joint Interoperability Test Command Records Management Application
Internet Request for Comments (RFC) Process
References

Much has been published on E-Discovery generally, so no list of references could hope to be complete. Here are a few papers that we know of that we believe would be useful as background reading for the focus of this workshop. Please send recommended additions for this list to oard-AT-umd.edu.
W. Andrews and D. Logan, Early Case Assessment: E-Discovery Beyond Judges and Regulators Is About Risks, Costs and Choices, January 27 (2010).
W. Andrews, D. Logan, J. Bace and S. Childs, E-Discovery SaaS and On-Premises Software Converge at Vendors as They Mature, July 29 (2010).
J. Bace, Communicating Metadata Issues in E-Discovery to the General Counsel, November 10 (2010).
J. Baron, Toward a Federal Benchmarking Standard For Evaluating Information Retrieval Products Used In E-Discovery, The Sedona Conference Journal, 6(1)237-246 (2005)
M. Grossman, Reflections from the Topic Authorities About the 2009 TREC Legal Track Interactive Task (2010).
M. Grossman and T. Sweeney, What lawyers need to know about search tools, National Law Journal, August (2010).
B. Hedin, S. Tomlinson, J. Baron and D. Oard TREC 2009 Track Overview (2010).
J. Krause, Human-Computer Assisted Search in EDD, Law Technology News, December 20 (2010).
D. Oard, J. Baron, B. Hedin, D. Lewis and S. Tomlinson, Evaluation of Information Retrieval for E-Discovery, Artificial Intelligence and Law 18(4)347-386 (2011).
D. Oard, J. Baron and D. Lewis, Some Lessons Learned to Date from the TREC Legal Track, 2006-2009, February 24 (2010)
V. Rangan, Why TREC Matters to Electronic Data Discovery, Law.com technology blog, April (2010).
H. Roitblat, A. Kershaw and P. Oot., Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, Journal of the American Society for Information Science and Technology, 61(1)70-80 (2010).
The Sedona Conference®, Commentary on Achieving Quality in E-Discovery (2009).
Organizing Committee

Jason R. Baron, National Archives and Records Administration, USA
Laura Ellsworth, Jones Day, USA
Dave Lewis, David D. Lewis Consulting, USA
Debra Logan, Gartner Research, UK
Douglas W. Oard, University of Maryland, USA