, ,

E-Discovery Process Improvement: The After-Action Audit

Process improvement involves an ongoing effort to identify what’s working well and identify what could be improved. In e-discovery, it’s sometimes painfully obvious when things didn’t work well, e.g., a production deadline is missed or sensitive data is produced. However, it’s not always obvious what could be improved – it’s hard to identify potential improvements if results are about what people thought was achievable.

After-action audits can be eye-opening in identifying ways to improve the e-discovery process. However, the term “improve” is rather broad; more specific goals will provide better guidance for the audit. Jeff Carr, long-time legal cost expert, recommends SMART goals – those that are Specific, Measurable, Achievable, Realistic, and Timely, e.g.:

  • Lower outside counsel document review fees by 20%
  • Lower hosting fees for e-discovery review platform vendors by 30%.
  • Identify cost-effective ways to get early looks at potential discovery from the very start of potential litigation.

One process I find useful is to select a case representative in complexity and scope to those ordinarily encountered by the client, and reprocess the same documents using alternative tool sets. Using actual case data has several advantages:

  • Proving scalability of alternative tools sets. Some tools look nice on small select demo data sets (does “Enron” sound familiar?) but don’t scale well for large collections.
  • Identifying “gotchas” in alternative tools. There can be idiosyncrasies in data sets that cause problems in some tools. Nothing identifies these problems like running actual client data.
  • Validating original technology. Search and analytics tools that performs similar functions may not produce the same results, e.g., some full text search software may have problems indexing specific document types. The audit provides a way to potentially identify weaknesses.

In the ideal world, there would be production notes detailing the tools that were used to achieve the original volume reduction, and the decisions that were made, and there would be bills from attorneys, review providers, hosting providers, and software providers. All that detail provides a baseline for comparison.

Audit Deliverables

The audit report should cover:

  • Alternative Techniques. What techniques and tools could have provided the same functionality in terms of eliminating irrelevant content and identifying relevant content, but at a lower cost? For example, social network analysis, key term logic testing, concept clustering, visual similarity, and other functions are available in a variety of software packages that can be provided without per-gigabyte processing fees.
  • Dollar Impact of Alternative Tools. Culling irrelevant content early in the process saves considerable money downstream, e.g., reduced hosting fees, and reduced attorney review time. The report can estimate those savings.
  • Recommended Training. What training should be provided to either make better use of existing tools or to use new tools?


The direct costs of the after-action audit needn’t be very large when the tools used for auditing are provided on a flat- or no-fee basis, i.e., not charged on a per-GB, per-user, or per-search basis. Most audits can be performed using low-cost cloud storage or existing consultant infrastructure.


Audits needn’t take a long time to complete. Large savings are usually quickly obvious, and useful, actionable data can be available within about a month.

Further Reading:


Eating at the E-Discovery Diner: Buffet or à la Carte?

There is a difference of opinion about the best way for corporations to buy e-discovery services. One view could be characterized as the “single-throat-to-choke” approach which focuses on accountability – the corporation wants to have a single party take complete responsibility for everything from collection through production so there’s no question who’s at fault if anything goes wrong. The other view is a more á la carte approach where the corporation buys services as needed from different providers.

My view is that the single-throat approach results in overpaying for e-discovery services. Corporations can obtain more cost-effective results by having consultants who specialize in using the most appropriate tools for the collection and initial culling and contract separately for the final review for the reduced data set. The corporation can retain full accountability by clearly delineated responsibilities and hand-offs between the two providers.

The collection-initial culling vendor is responsible for gathering initial content and applying early analytics and other tools to the content while keeping detailed logs of what was done, what tools were used, and what culling decisions were made.

In the shared responsibility model, the client specifies the format of files that will be handed over to the final review vendor as well as the method and date of delivery. The final review vendor is then tasked with documenting the steps taken to further cull the collection as well as the delivery date and method of production, including generating privilege logs.

The single vendor approach is like eating every meal and taking every coffee break at an all-you-can-eat buffet. You overpay and consume too much of the wrong things. As a matter of fact, there are many early analytics tools that give early insight into potentially-responsive documents while avoiding the per GB cost model used by most integrated approach vendors. Every GB that is screened out before going to final review can save the corporate client hundreds of dollars per year.

Final review vendors typically have large staffs for help desk, technical support, consulting, and sales personnel, and have major investments in licenses, processing infrastructure, advertising, trade shows, and office space. All those expenses have to be covered to stay in business. By contrast, discovery boutiques specializing in collection and initial culling can be nimbler and offer different pricing models for delivering a comparable or an extended range of early discovery analytics tools.

Note that time and data security are major considerations when deciding what approach to take when contracting for e-discovery services. Many early analytics tools can be deployed and yield results in the time it can take to setup a final review platform, administer passwords, conduct training, and begin to load the initial data. Furthermore, from a data security standpoint, it is much better to screen out as much content as possible before putting content on a final review platform where dozens of people will have access to some or all of it.


What Are In-House Early Case Analytics?

In-house early analytics are discovery intelligence gathering and reporting mechanisms that help in-house counsel and outside counsel understand a corpus of potentially-relevant documents and e-mail.

In-house early case analytics gives counsel the ability to make well-informed decisions about what documents and e-mail are clearly non-relevant so that these files can be removed prior to transfer to outside counsel for traditional review.

Said more specifically, the purpose of in-house early analytics is to educate and inform counsel as to the nature, scope and potential size of the document request.  In many legal cases, outside counsel is oblivious as to the size of burden a discovery request places upon a company.  In-house early analytics brings transparency to outside counsel so that they can refine the request.  Meanwhile, in-house early case analytics informs managing counsel as to the actual costs of discovery – prior to ESI being sent out the door.

In-house early analytics come to counsel in the form of informational reports and visualizations, three of which I list here:

  • Key term hit reports by custodian (see example below)

    Custodian Analysis by Term

  • Visual charts and graphs (concept maps, conversation clusters, clusters of similar documents, etc)

    • Concept cluster maps (visualization that clusters similar documents together)Concept Cluster Maps
    • Conversation cluster maps (a visualization that shows the e-mail communications)Conversion Cluster Maps
    • Interactive screen share sessions where outside counsel is able to view a file share firsthand.


In my personal experience, when outside counsel is educated and informed via early in-house analytics, they will often then have sufficient information necessary to refine and further perfect their key terms list.  This refinement will often have a significant impact on the total number of documents that end up in traditional attorney review.