Technology Advantage

Technology Assisted Review: Or, How I Stopped Worrying and Learned to Love a Computer Program (PART ONE)

Russell Beets Amy Catton May 16th, 2018
TAR

PART ONE: This is part one of a series on my journey to appreciating TAR. Be on the lookout for part two in the coming weeks.

Recently, I (Russ Beets) began work on a complex litigation case that had millions of documents to review with many moving parts and quick deadlines that made completing assignments daunting, to say the least. We were divided into teams to tackle different aspects of the review (e.g., first level review for production; QC for privilege and privilege logging; preparation for custodian depositions; preparation of evidence to support our case theories; etc.). My team was tasked with locating documents that would help tell our side of the story and provide evidence for our case theories. We determined that simply running targeted searches to find this evidence was not the best approach, in part because the issues were broadly defined and had multiple subparts, and in part because of the sheer number of documents in the database (over 2.5 million records). As an alternative, we decided to utilize technology assisted review or “TAR” (also known in the industry as predictive coding). Even though I have practiced law for less than 20 years (and don’t consider myself to be “old” per se), I will admit to being a bit “old-school” when it comes to assisted review – likely out of nothing more than a misplaced fear that these types of technologies would make my job obsolete. However, after learning more about the process and using TAR for several weeks in a row, I came to the realization that it is meant to assist attorneys and streamline review, and is certainly not my replacement.

Amy Catton and Clara Skorstad, two Senior Project Managers on my team, are seasoned experts in TAR and helped with the drafting of this blog. Amy managed my most recent TAR project and Clara is a member of Duke University’s EDRM Technology Assisted Review Project Team, working to develop best practices. If anything below sounds like it came from a technical expert, it likely came from one or both of them.

What is TAR?

TAR is a concept-based method of document coding that leverages machine-learning techniques with the input of human reviewers to automate the review process. In TAR, you are able to leverage a small review team and statistical sampling against large volumes of data to propagate coding to unreviewed documents. Human reviewers, based on their own coding decisions, train the system to recognize documents that are likely to be relevant and likely to be irrelevant. In addition to training, the system conducts quality control rounds to ensure the confidence level (a measurement of quality control) is high. Most courts that have addressed TAR agree that it provides accurate and consistent coding calls, and can be even more reliable than human review when it comes to determining whether a document is likely to be relevant.  

How Does TAR Work?

  1. Legal professionals review and code a subset of data from the overall collection (known as the “seed set”), typically for responsiveness.
  2. The software then compares the human coding against each document’s content, determining the criteria that make a document more likely to be relevant.
  3. An algorithm built into the technology then applies the reviewer’s logic to classify documents across the data collection as responsive or not responsive.
  4. As reviewers feed additional coded documents into the system (or “train” the computer), the technology refines its decision-making ability (or “learns” what is relevant) and the accuracy and defensibility of the process increase. In other words, the human reviewer and the software work collaboratively to refine the set of responsive documents and reach what is known as “stabilization.”

When Should You Consider Using TAR?

While TAR is an amazing tool to help assist with document review, it may not make sense in all cases, and in fact traditional review is sometimes the better option. Below is a set of factors to consider when determining whether TAR is a viable option in your particular case:

  • Number of Documents

Generally speaking, TAR programs need a lot of data in order to work properly and effectively. For example, we generally would not recommend trying TAR with fewer than 50,000 documents. Simply put, the program is just more effective when there are more documents for the system to analyze. Aside from the effectiveness of the tool, there is a significant amount of time needed to set up the TAR job, review a training set of documents, conduct a QC round and continue those steps until the system has reached stabilization. The TAR workflow is different than the traditional document review workflow and often times utilizes more costly attorney resources to train the system.  If the set is less than 50,000 documents then the time to complete these steps may outweigh the benefits.  In the case of a smaller data set, it probably makes sense to utilize other analytics tools and conduct a more traditional review.

  • Stage of the Litigation Process

Another factor to consider is the current stage of the litigation. Because TAR can best be thought of as a learning tool, it makes more sense to utilize it at the onset of litigation and document collection/review. As will be discussed below in the section regarding TAR processes, it is beneficial to use your case expert(s) to review documents and teach the system, instead of having contract reviewers sifting through documents somewhat blindly and passing them up the chain to more senior attorneys for further review. Although this will likely take some time at the onset to get the system sufficiently trained, it will, down the road, lead to a better tailored set of documents for review.

  • Cost

Cost is generally not a major factor in whether to use TAR with large-scale reviews, as the use of such workflows is likely already contained within whichever software program you are using (although some vendors do charge a per document or per GB rate to use TAR tools). Regardless, when TAR is properly implemented, any costs associated with management of TAR should be made up through cost savings during the review timeline.

  • Other Uses of TAR

It is helpful when considering whether to use TAR to understand that it is not an all or nothing proposition. As I discovered during my own TAR review, it is actually a supplement to traditional review rather than a complete replacement of it - helping to get the most important, relevant documents in front of the reviewers. 

Part two of this series will address the TAR process, advantages and disadvantages of TAR, and my current thoughts on using the technology. Stay tuned!

 

DISCLAIMER: The information contained in this blog is not intended as legal advice or as an opinion on specific facts. For more information about these issues, please contact the author(s) of this blog or your existing LitSmart contact. The invitation to contact the author is not to be construed as a solicitation for legal work. Any new attorney/client relationship will be confirmed in writing.

Topics: E-Discovery Technology Assisted Review TAR Predictive Coding Relativity Assisted Review Analytics E-Discovery Technology

Subscribe to the E-Discovery Newsletter

Related Posts

  • Data Mapping - Why is it Important for Successful E-Discovery?

    A data map is usually something that is generated along with a data retention plan and is a great starting point if and when an organization becomes involved in litigation and discovery. Long before litigation and discovery, however, it is important to understand the relevant data sources that may be in play and where those can be found within the organization’s various data systems. This blog explains what a data map is, what it contains and how it is helpful.

  • Pitfalls of Complex Search Protocols in ESI Agreements

    ESI Agreements cover the full gambit of e-discovery issues, from preservation expectations to production specifications. Sometimes, these agreements include an overview of the process by which the parties will identify the universe of potentially responsive documents by using specific date ranges, identifying priority custodians and developing proposed search terms. Adding another layer of complexity, this discussion is often had in a vacuum before the parties even know how much data their clients have. Coming up with criteria that identifies the relevant documents, but is not overly broad, in this vacuum can be difficult. However, the process is important because the date range, custodians and terms will inevitably dictate how much data is collected, processed and reviewed, which can significantly affect any litigation budget.

Newest Posts

  • When The Timing of Your Spoliation Motion Can Be As Important As Its Substance

    A motion for an adverse inference was denied in Pratt v. Robbins, et al. where Defendants failed to preserve or produce a video that might have contained pivotal evidence going to the heart Plaintiff’s civil rights claim for excessive force.  Plaintiff argued that Defendants spoliated evidence by failing to produce the video footage that may have recorded the use of force at issue.   A party seeking spoliation sanctions bears the burden of proving all of the elements of Rule 37(e), and under Fourth Circuit precedent is generally held to a clear and convincing standard. Plaintiff’s decision to suddenly cry foul on the eve of trial did not go over well with the Court.

  • PLAY IT AGAIN, SAM . . . BUT WAIT – WHAT IS HE PLAYING, AND WHO IS SAM?: Website Session Replay Software and Wiretapping

    The trend of class action cases alleging wiretap statute violations against website operators for the use of session replay software to collect information of website users continues to be a concern for companies in 2024.  While many of these cases have turned on whether website operators disclosed and obtained consent from website users prior to deploying any session replay software, a recent decision in a pending class action brought under the Pennsylvania Wiretap Act indicates that consent alone may not be the end of the story in these cases.

  • Utilizing E-Discovery Tools in Innovative Ways

    As part of the e-discovery process, we use digital means to identify relevant information for use in a legal proceeding. The proceeding may be a large-scale medical malpractice lawsuit, a patent infringement case, a government investigation, or countless other legal actions. However, the skills and resources used in those types of matters can also be used in other areas.