Technology Advantage

Technology Assisted Review: Or, How I Stopped Worrying and Learned to Love a Computer Program (PART ONE)


PART ONE: This is part one of a series on my journey to appreciating TAR. Be on the lookout for part two in the coming weeks.

Recently, I (Russ Beets) began work on a complex litigation case that had millions of documents to review with many moving parts and quick deadlines that made completing assignments daunting, to say the least. We were divided into teams to tackle different aspects of the review (e.g., first level review for production; QC for privilege and privilege logging; preparation for custodian depositions; preparation of evidence to support our case theories; etc.). My team was tasked with locating documents that would help tell our side of the story and provide evidence for our case theories. We determined that simply running targeted searches to find this evidence was not the best approach, in part because the issues were broadly defined and had multiple subparts, and in part because of the sheer number of documents in the database (over 2.5 million records). As an alternative, we decided to utilize technology assisted review or “TAR” (also known in the industry as predictive coding). Even though I have practiced law for less than 20 years (and don’t consider myself to be “old” per se), I will admit to being a bit “old-school” when it comes to assisted review – likely out of nothing more than a misplaced fear that these types of technologies would make my job obsolete. However, after learning more about the process and using TAR for several weeks in a row, I came to the realization that it is meant to assist attorneys and streamline review, and is certainly not my replacement.

Amy Catton and Clara Skorstad, two Senior Project Managers on my team, are seasoned experts in TAR and helped with the drafting of this blog. Amy managed my most recent TAR project and Clara is a member of Duke University’s EDRM Technology Assisted Review Project Team, working to develop best practices. If anything below sounds like it came from a technical expert, it likely came from one or both of them.

What is TAR?

TAR is a concept-based method of document coding that leverages machine-learning techniques with the input of human reviewers to automate the review process. In TAR, you are able to leverage a small review team and statistical sampling against large volumes of data to propagate coding to unreviewed documents. Human reviewers, based on their own coding decisions, train the system to recognize documents that are likely to be relevant and likely to be irrelevant. In addition to training, the system conducts quality control rounds to ensure the confidence level (a measurement of quality control) is high. Most courts that have addressed TAR agree that it provides accurate and consistent coding calls, and can be even more reliable than human review when it comes to determining whether a document is likely to be relevant.  

How Does TAR Work?

  1. Legal professionals review and code a subset of data from the overall collection (known as the “seed set”), typically for responsiveness.
  2. The software then compares the human coding against each document’s content, determining the criteria that make a document more likely to be relevant.
  3. An algorithm built into the technology then applies the reviewer’s logic to classify documents across the data collection as responsive or not responsive.
  4. As reviewers feed additional coded documents into the system (or “train” the computer), the technology refines its decision-making ability (or “learns” what is relevant) and the accuracy and defensibility of the process increase. In other words, the human reviewer and the software work collaboratively to refine the set of responsive documents and reach what is known as “stabilization.”

When Should You Consider Using TAR?

While TAR is an amazing tool to help assist with document review, it may not make sense in all cases, and in fact traditional review is sometimes the better option. Below is a set of factors to consider when determining whether TAR is a viable option in your particular case:

  • Number of Documents

Generally speaking, TAR programs need a lot of data in order to work properly and effectively. For example, we generally would not recommend trying TAR with fewer than 50,000 documents. Simply put, the program is just more effective when there are more documents for the system to analyze. Aside from the effectiveness of the tool, there is a significant amount of time needed to set up the TAR job, review a training set of documents, conduct a QC round and continue those steps until the system has reached stabilization. The TAR workflow is different than the traditional document review workflow and often times utilizes more costly attorney resources to train the system.  If the set is less than 50,000 documents then the time to complete these steps may outweigh the benefits.  In the case of a smaller data set, it probably makes sense to utilize other analytics tools and conduct a more traditional review.

  • Stage of the Litigation Process

Another factor to consider is the current stage of the litigation. Because TAR can best be thought of as a learning tool, it makes more sense to utilize it at the onset of litigation and document collection/review. As will be discussed below in the section regarding TAR processes, it is beneficial to use your case expert(s) to review documents and teach the system, instead of having contract reviewers sifting through documents somewhat blindly and passing them up the chain to more senior attorneys for further review. Although this will likely take some time at the onset to get the system sufficiently trained, it will, down the road, lead to a better tailored set of documents for review.

  • Cost

Cost is generally not a major factor in whether to use TAR with large-scale reviews, as the use of such workflows is likely already contained within whichever software program you are using (although some vendors do charge a per document or per GB rate to use TAR tools). Regardless, when TAR is properly implemented, any costs associated with management of TAR should be made up through cost savings during the review timeline.

  • Other Uses of TAR

It is helpful when considering whether to use TAR to understand that it is not an all or nothing proposition. As I discovered during my own TAR review, it is actually a supplement to traditional review rather than a complete replacement of it - helping to get the most important, relevant documents in front of the reviewers. 

Part two of this series will address the TAR process, advantages and disadvantages of TAR, and my current thoughts on using the technology. Stay tuned!


DISCLAIMER: The information contained in this blog is not intended as legal advice or as an opinion on specific facts. For more information about these issues, please contact the author(s) of this blog or your existing LitSmart contact. The invitation to contact the author is not to be construed as a solicitation for legal work. Any new attorney/client relationship will be confirmed in writing.

Topics: E-Discovery Technology Assisted Review TAR Predictive Coding Relativity Assisted Review Analytics E-Discovery Technology

Subscribe to the E-Discovery Newsletter