Technology Advantage

The Growing Source of ESI: Structured Data and Messaging Platforms

David Welch June 13th, 2019
Structured Data

Structured vs. Unstructured Data

It takes a minute to come to terms with the definitions of "structured" and "unstructured" data. It seems logical to associate “formatting,” such as the formatting that comes with word documents (indentation, headers and footers, paragraphs, etc.) with “structure,” but the terms are used very differently when it comes to describing data. Structured data traditionally exists in relational databases such as SQL and more simply, Excel.  Unstructured data refers to data that does not easily fit into databases and includes Word documents, emails and PDFs. It seems counter-intuitive, but data that is formatted in a way that makes sense for everyday communication is actually unstructured data. 

Structured vs. Unstructured Data in E-Discovery

While structured data is easy for a computer or data analyst to interpret, it presents challenges in e-discovery. Unstructured data, on the other hand, is something that we have long since come to terms with in e-discovery. The development of software platforms that allow us to view, search, code and analyze unstructured data is pretty amazing. Predictive coding is already giving way to advanced AI, in order to keep up with expanding volume of data in discovery. But these tools, and even simple tools such as text searching, struggle to provide a means to handle structured data. 

While the task may seem daunting, plenty of effort has been aimed at handling structured data. Even the most basic form of structured data - Excel - presents challenges. Imaging doesn’t make sense because of exorbitant page counts, so we generally produce Excel files natively, which presents its own challenges, such as redacting privileged and personal information. However, tools have been, and continue to be, developed to meet these challenges. 

The New Structured Data Source

In the last few years, we have seen the emergence of a new structured data source - messaging platforms. Instant communication platforms such as Skype, Slack and Bloomberg have exploded in the corporate world and, as a result, have created a new challenge in e-discovery. These challenges run the throughout the EDRM from identification, preservation and collection all the way through production. 

Challenge # 1: Identifying the Data

One of the most basic challenges comes in the form of simply identifying a custodian. For the sake of this blog I will use Slack as an example, but the lessons can generally be extended to other messaging platforms as well. A custodian is generally considered to be the person who has control over an electronic document, with an email box being the most common example (for example, I am the custodian of my email). With a messaging platform, however, who the custodian is may not be so straight forward. There are public channels, private channels, private messages and so on. An individual participating in a public channel does not have any control over local storage of the data, so are they a custodian? I think most of us would agree that by participating in a relevant communication, their data is discoverable, but how do we identify and collect that user’s data?

Challenge # 2: Preserving the Data

Preservation becomes an enormous factor in what is inevitably available to be collected. The default setting in Slack is to preserve all data forever. However, an administrator can alter these settings to limit retention to a certain time frame or allow individuals to set their own retention policies for their private channels. With such a framework, on the one hand, we run the risk of having an uncontrollable amount of data; on the other hand, we run the risk of an individual user deleting complete channels of communication (creating just the opposite problem). It is critical that we are aware of these options and the implications.

Challenge # 3: Collecting the Data

When it comes to data collection, exporting data from Slack requires administrative control. While it is convenient to have somewhat centralized control of any data for collection, it presents problems when the collection could be limited in scope to only certain individual custodians.  Participation by a user in public channel communication will require the export of that entire channel, which may lead to the collection of an enormous amount of data that is not relevant, which in turn creates challenges for culling and review.

When it comes to exporting the data, all messaging platforms have unique export formats, and Slack data can be particularly challenging. Slack exports to a zip file with channels stored as .json files (JavaScript Object Notation). I’ll show an example shortly, but the gist is that this is computer code. A comparison might be an email message that is created in .html. The .html contains the formatting that allows a web browser to display email messages in the easily readable format we are used to. The same is true of Slack messages - with one big difference.  Slack data is formatted within Slack, but does not export to a format that can easily be re-constituted to its visual format, so you just get the code.

Challenge # 4: Making Sense of the Data

What does Slack data look like? Here is an example of a simple Slack .json file:

{

        "user": "user1",

        "type": "message",

        "subtype": "channel_leave",

        "ts": "1526494600.000712",

        "text": "<@user1> has left the channel"

      }

In this example, all we find out is that a particular user ‘has left the channel.’ Imagine how much data can be generated if it takes that much text just to note that someone has joined or left a channel. Here is an example of a brief conversation:

    {

      "date": "September 13th, 2017",

      "time": "3:20 PM",

      "sender": "user1",

      "message": "yeah"

    },

    null,

    null,

    {

      "date": "September 15th, 2017",

      "time": "4:12 PM",

      "sender": "user2",

      "message": "Can you send me an invite to test flight?"

    },

    {

      "date": "September 15th, 2017",

      "time": "4:22 PM",

      "sender": "user1",

      "message": "yep. what's your user ID?"

    },

 

All of that just to convey the conversation:

yeah

Can you send me an invite to test flight?

yep. what's your user ID?

 

This particular snippet came from a channel that generated over 1200 pages of text and an average of only 10 lines of actual communication per page. When ultimately re-formatted, the entire communication was reduced to 200 pages. Still daunting, but a reduction in review that would be substantial.   

Challenge #5: Reviewing and Producing the Data

The last line of this message, "yep. what's your user ID?", creates another problem for review and production, namely how do we handle privileged or private information? Producing the files natively would be a reasonable solution if we didn’t have privacy concerns. The native .json files open readily enough in text editors, but the page count is unwieldy if you have to image in order redact sensitive data. 

Aside from the formatting issue related to text, another concern is how to handle non-text communication within messaging platforms, such as emojis. There isn’t any denying that such communication is part of our vocabulary and now we need to identify ways to deal with it. A  is a perfectly understood response to question posed in Slack or Skype, but how will it be handled when it is formatted like this after collection?

“thumb_800_w”: 800,

“thumb_800_h”: 301,

“image_exif_rotation”: 1,

“original_w”: 953,

“original_h”: 358,   

Where Do We Go From Here?

Even this cursory look at some of the structured data coming out of messaging platforms can be a little unnerving. But don’t worry, there are already a number of solutions out there. E-discovery tools have been developed to handle data coming from various messaging platforms. While no one has developed a comprehensive solution for all messaging platforms as of yet, nearly every platforms has been addressed by someone.

There are a number of approaches to representing structured data from messaging platforms in reviewable formats. One tool on the market parses the structured data and re-formats as .html that can be viewed in a browser or uploaded into review platforms along with metadata load files. One of the examples from above is now represented in this format:

 

This format is certainly easier to review and it provides reasonable imaging options for production and the redaction of sensitive information. While this example only involves text, images and emojis have also been handled by tools in the market. Some tools that process Slack data are able to represent emojis much as you would see them in Slack. Others choose to keep emojis in Slack text format, making the argument that the text format would allow for text searching of a specific emoji. Here is an example: 

 

Whether making emojis searchable might be useful is probably case specific, but it is an interesting approach and highlights the differences in handling structured data sources.

As with most anything in e-discovery, preparation and forethought are crucial for success. Now that we understand the potential complexities of structured data, we can imagine how important it would be to discuss structured data sources with our clients early to get ahead of any potential issues down the line. So, as take-aways:

  1. Discuss with clients data retention and preservation policies and practices with respect to structured data sources, including messaging platforms.
  2. Incorporate structured data sources into discussions with opposing counsel and consider specifically addressing them in your ESI agreements (including whether they are subject to preservation and production at all).
  3. Identify internal or external resources for the collection and processing of structured data and get an idea of what the reviewable format will be beforehand.

These lessons are not different from how we approach other sources of ESI. It does take a little extra effort because structured data coming from messaging platforms is evolving quickly and it takes time for the tools designed for the legal community to catch up. So it is time to start thinking about this data, because sooner or later it is coming to a case near you.

 

DISCLAIMER: The information contained in this blog is not intended as legal advice or as an opinion on specific facts. For more information about these issues, please contact the author(s) of this blog or your existing LitSmart contact. The invitation to contact the author is not to be construed as a solicitation for legal work. Any new attorney/client relationship will be confirmed in writing.

Topics: E-Discovery Structured Data Unstructured Data Data Processing Data Collection Document Review

Subscribe to the E-Discovery Newsletter

Related Posts

  • Data Mapping - Why is it Important for Successful E-Discovery?

    A data map is usually something that is generated along with a data retention plan and is a great starting point if and when an organization becomes involved in litigation and discovery. Long before litigation and discovery, however, it is important to understand the relevant data sources that may be in play and where those can be found within the organization’s various data systems. This blog explains what a data map is, what it contains and how it is helpful.

  • Pitfalls of Complex Search Protocols in ESI Agreements

    ESI Agreements cover the full gambit of e-discovery issues, from preservation expectations to production specifications. Sometimes, these agreements include an overview of the process by which the parties will identify the universe of potentially responsive documents by using specific date ranges, identifying priority custodians and developing proposed search terms. Adding another layer of complexity, this discussion is often had in a vacuum before the parties even know how much data their clients have. Coming up with criteria that identifies the relevant documents, but is not overly broad, in this vacuum can be difficult. However, the process is important because the date range, custodians and terms will inevitably dictate how much data is collected, processed and reviewed, which can significantly affect any litigation budget.

Newest Posts

  • Now You See Me, Now You Don’t: Ephemeral Messaging Challenges

    The emergence of ephemeral messaging applications to communicate with friends, family, and coworkers quickly, securely, and effortlessly has boomed over the past decade. In that time, users of ephemeral messaging apps have risen significantly, from millions to billions of active users! Ephemeral messaging is an integral and evolving part of both individual and company communications with advantages and challenges impacting companies and the legal industry. Companies must consider the ramifications of their employees' use of ephemeral messaging and adopt policies and procedures to best protect themselves and comply with requirements relating to litigation and regulation. Courts have already begun addressing the failure to preserve relevant ephemeral messages and have been issuing sanctions in a myriad of legal challenges. Ephemeral messaging is here to stay, and companies, courts, and attorneys must understand and adapt to evolve with this emerging technology.

  • Generative AI E-Discovery Tools and the Importance of Prompt Engineering

    The explosion of LLMs (large language models) and other Generative AI tools designed to increase workplace efficiency and productivity has created a new lexicon of jargon and definitions. The term “Prompt Engineering” may be the most widely used new term to describe the method of using LLMs and Generative AI tools. Prompt engineering is the creation of text-based instructions or cues that a person uses to direct LLMs or other generative AI tools. These instructions are then interpreted by the AI tool. Learning to harness the power of e-discovery AI Tools like Relativity’s aiR for Review through prompts specifically constructed to produce the output you desire in your case can increase your efficiency. Implementing these prompt creation and iteration tips and tricks will help you use Relativity’s aiR for Review tool to more effectively to assist with litigation, document review, and deposition or trial preparation.

  • Solving the “‘Privilege Log’ Problem”: Proposed Changes to Federal Rules of Civil Procedure 16 & 26

    Changes to Federal Rules of Civil Procedure 16(b) and 26(f) are projected to come into effect in December 2025 that have been promulgated to address what proponents refer to as the “‘privilege log’ problem.” December 2025 is still relatively far in the future and whether the proposed amendments to Rules 16(b) and 26(f) are actually adopted is still subject to additional approvals, including approval by the Supreme Court. Nevertheless, bear in mind that nothing in the current Federal Rules of Civil Procedure prevents the parties from implementing the more proactive approach the proposed amendments seek to achieve.