Technology Advantage

Technology Assisted Review: Or, How I Stopped Worrying and Learned to Love a Computer Program (PART ONE)

Russell Beets Amy Catton May 16th, 2018
TAR

PART ONE: This is part one of a series on my journey to appreciating TAR. Be on the lookout for part two in the coming weeks.

Recently, I (Russ Beets) began work on a complex litigation case that had millions of documents to review with many moving parts and quick deadlines that made completing assignments daunting, to say the least. We were divided into teams to tackle different aspects of the review (e.g., first level review for production; QC for privilege and privilege logging; preparation for custodian depositions; preparation of evidence to support our case theories; etc.). My team was tasked with locating documents that would help tell our side of the story and provide evidence for our case theories. We determined that simply running targeted searches to find this evidence was not the best approach, in part because the issues were broadly defined and had multiple subparts, and in part because of the sheer number of documents in the database (over 2.5 million records). As an alternative, we decided to utilize technology assisted review or “TAR” (also known in the industry as predictive coding). Even though I have practiced law for less than 20 years (and don’t consider myself to be “old” per se), I will admit to being a bit “old-school” when it comes to assisted review – likely out of nothing more than a misplaced fear that these types of technologies would make my job obsolete. However, after learning more about the process and using TAR for several weeks in a row, I came to the realization that it is meant to assist attorneys and streamline review, and is certainly not my replacement.

Amy Catton and Clara Skorstad, two Senior Project Managers on my team, are seasoned experts in TAR and helped with the drafting of this blog. Amy managed my most recent TAR project and Clara is a member of Duke University’s EDRM Technology Assisted Review Project Team, working to develop best practices. If anything below sounds like it came from a technical expert, it likely came from one or both of them.

What is TAR?

TAR is a concept-based method of document coding that leverages machine-learning techniques with the input of human reviewers to automate the review process. In TAR, you are able to leverage a small review team and statistical sampling against large volumes of data to propagate coding to unreviewed documents. Human reviewers, based on their own coding decisions, train the system to recognize documents that are likely to be relevant and likely to be irrelevant. In addition to training, the system conducts quality control rounds to ensure the confidence level (a measurement of quality control) is high. Most courts that have addressed TAR agree that it provides accurate and consistent coding calls, and can be even more reliable than human review when it comes to determining whether a document is likely to be relevant.  

How Does TAR Work?

  1. Legal professionals review and code a subset of data from the overall collection (known as the “seed set”), typically for responsiveness.
  2. The software then compares the human coding against each document’s content, determining the criteria that make a document more likely to be relevant.
  3. An algorithm built into the technology then applies the reviewer’s logic to classify documents across the data collection as responsive or not responsive.
  4. As reviewers feed additional coded documents into the system (or “train” the computer), the technology refines its decision-making ability (or “learns” what is relevant) and the accuracy and defensibility of the process increase. In other words, the human reviewer and the software work collaboratively to refine the set of responsive documents and reach what is known as “stabilization.”

When Should You Consider Using TAR?

While TAR is an amazing tool to help assist with document review, it may not make sense in all cases, and in fact traditional review is sometimes the better option. Below is a set of factors to consider when determining whether TAR is a viable option in your particular case:

  • Number of Documents

Generally speaking, TAR programs need a lot of data in order to work properly and effectively. For example, we generally would not recommend trying TAR with fewer than 50,000 documents. Simply put, the program is just more effective when there are more documents for the system to analyze. Aside from the effectiveness of the tool, there is a significant amount of time needed to set up the TAR job, review a training set of documents, conduct a QC round and continue those steps until the system has reached stabilization. The TAR workflow is different than the traditional document review workflow and often times utilizes more costly attorney resources to train the system.  If the set is less than 50,000 documents then the time to complete these steps may outweigh the benefits.  In the case of a smaller data set, it probably makes sense to utilize other analytics tools and conduct a more traditional review.

  • Stage of the Litigation Process

Another factor to consider is the current stage of the litigation. Because TAR can best be thought of as a learning tool, it makes more sense to utilize it at the onset of litigation and document collection/review. As will be discussed below in the section regarding TAR processes, it is beneficial to use your case expert(s) to review documents and teach the system, instead of having contract reviewers sifting through documents somewhat blindly and passing them up the chain to more senior attorneys for further review. Although this will likely take some time at the onset to get the system sufficiently trained, it will, down the road, lead to a better tailored set of documents for review.

  • Cost

Cost is generally not a major factor in whether to use TAR with large-scale reviews, as the use of such workflows is likely already contained within whichever software program you are using (although some vendors do charge a per document or per GB rate to use TAR tools). Regardless, when TAR is properly implemented, any costs associated with management of TAR should be made up through cost savings during the review timeline.

  • Other Uses of TAR

It is helpful when considering whether to use TAR to understand that it is not an all or nothing proposition. As I discovered during my own TAR review, it is actually a supplement to traditional review rather than a complete replacement of it - helping to get the most important, relevant documents in front of the reviewers. 

Part two of this series will address the TAR process, advantages and disadvantages of TAR, and my current thoughts on using the technology. Stay tuned!

 

DISCLAIMER: The information contained in this blog is not intended as legal advice or as an opinion on specific facts. For more information about these issues, please contact the author(s) of this blog or your existing LitSmart contact. The invitation to contact the author is not to be construed as a solicitation for legal work. Any new attorney/client relationship will be confirmed in writing.

Topics: E-Discovery Technology Assisted Review TAR Predictive Coding Relativity Assisted Review Analytics E-Discovery Technology

Subscribe to the E-Discovery Newsletter

Related Posts

  • Data Mapping - Why is it Important for Successful E-Discovery?

    A data map is usually something that is generated along with a data retention plan and is a great starting point if and when an organization becomes involved in litigation and discovery. Long before litigation and discovery, however, it is important to understand the relevant data sources that may be in play and where those can be found within the organization’s various data systems. This blog explains what a data map is, what it contains and how it is helpful.

  • Pitfalls of Complex Search Protocols in ESI Agreements

    ESI Agreements cover the full gambit of e-discovery issues, from preservation expectations to production specifications. Sometimes, these agreements include an overview of the process by which the parties will identify the universe of potentially responsive documents by using specific date ranges, identifying priority custodians and developing proposed search terms. Adding another layer of complexity, this discussion is often had in a vacuum before the parties even know how much data their clients have. Coming up with criteria that identifies the relevant documents, but is not overly broad, in this vacuum can be difficult. However, the process is important because the date range, custodians and terms will inevitably dictate how much data is collected, processed and reviewed, which can significantly affect any litigation budget.

Newest Posts

  • Now You See Me, Now You Don’t: Ephemeral Messaging Challenges

    The emergence of ephemeral messaging applications to communicate with friends, family, and coworkers quickly, securely, and effortlessly has boomed over the past decade. In that time, users of ephemeral messaging apps have risen significantly, from millions to billions of active users! Ephemeral messaging is an integral and evolving part of both individual and company communications with advantages and challenges impacting companies and the legal industry. Companies must consider the ramifications of their employees' use of ephemeral messaging and adopt policies and procedures to best protect themselves and comply with requirements relating to litigation and regulation. Courts have already begun addressing the failure to preserve relevant ephemeral messages and have been issuing sanctions in a myriad of legal challenges. Ephemeral messaging is here to stay, and companies, courts, and attorneys must understand and adapt to evolve with this emerging technology.

  • Generative AI E-Discovery Tools and the Importance of Prompt Engineering

    The explosion of LLMs (large language models) and other Generative AI tools designed to increase workplace efficiency and productivity has created a new lexicon of jargon and definitions. The term “Prompt Engineering” may be the most widely used new term to describe the method of using LLMs and Generative AI tools. Prompt engineering is the creation of text-based instructions or cues that a person uses to direct LLMs or other generative AI tools. These instructions are then interpreted by the AI tool. Learning to harness the power of e-discovery AI Tools like Relativity’s aiR for Review through prompts specifically constructed to produce the output you desire in your case can increase your efficiency. Implementing these prompt creation and iteration tips and tricks will help you use Relativity’s aiR for Review tool to more effectively to assist with litigation, document review, and deposition or trial preparation.

  • Solving the “‘Privilege Log’ Problem”: Proposed Changes to Federal Rules of Civil Procedure 16 & 26

    Changes to Federal Rules of Civil Procedure 16(b) and 26(f) are projected to come into effect in December 2025 that have been promulgated to address what proponents refer to as the “‘privilege log’ problem.” December 2025 is still relatively far in the future and whether the proposed amendments to Rules 16(b) and 26(f) are actually adopted is still subject to additional approvals, including approval by the Supreme Court. Nevertheless, bear in mind that nothing in the current Federal Rules of Civil Procedure prevents the parties from implementing the more proactive approach the proposed amendments seek to achieve.