Opinion
No. 16 CV 10156
09-03-2020
JENNIFER LIVINGSTON, et al., Plaintiffs, v. THE CITY OF CHICAGO, Defendant.
MEMORANDUM OPINION and ORDER
Before the court is Plaintiffs' motion to compel the City of Chicago (the "City") to use a particular methodology for identifying responsive ESI. For the following reasons, the motion is denied:
Background
Discovery in this 2016 case did not get underway until mid-2019 when nearly two years of settlement negotiations came to a head. (R. 89; R. 147.) While the parties successfully resolved some issues, many issues remain in dispute. The court has since ruled on a number of discovery disputes, and the current motion picks up where the parties' most recent quarrel left off. Starting in May 2019 the parties could not agree on the method to be employed for collecting and searching the City's ESI. Plaintiffs proposed that an outside vendor first export the emails and then perform keyword searches to identify the initial universe of emails, while the City wanted to use its own Microsoft Tool to perform a simple search prior to exporting any data. Plaintiffs also wanted the City to produce all of the emails identified through keyword searches without any further review for responsiveness and for privilege. In September 2019 Plaintiffs filed a motion asking the court to adopt their protocol and the parties fully briefed the issues. (R. 209; R. 219; R. 226.)
On November 20, 2019, the court entered an order granting in part and denying in part Plaintiffs' motion. (R. 239.) As part of this order, the court required the City to retain an outside vendor to export emails dated from July 1, 2014, to September 3, 2015, and then apply an initial keyword search using Plaintiffs' search terms. (Id.) The court noted that "[d]epending on the number of hits after the initial keyword search using Plaintiffs' proposal, the parties may use more finite terms to reduce the number of hits." (Id.) The court further rejected Plaintiffs' request that once the initial universe of emails had been identified through keyword searches, the City should produce the same without any further review. (Id.)
At a status conference in April 2020, the City reported that the emails had been collected and searched, resulting in 192,000 unique emails or a total of approximately 1.3 million pages of documents. (R. 286.) The City informed the court that it intended to use technology-assisted review ("TAR") to identify relevant responsive documents to be produced from this ESI collection. (Id.) Upon hearing this, Plaintiffs expressed concern that TAR would exclude responsive documents from the review process. (Id.) In their view, the City's use of TAR to conduct its responsiveness review is inconsistent with the court's November 2019 order. Plaintiffs filed the current motion for compliance with the order or, in the alternative, for entry of their proposed TAR protocol. (R. 289.)
Analysis
In their motion for compliance, Plaintiffs seek an order directing the City to use agreed-upon search terms to identify responsive documents and then to perform a manual review for privilege. (R. 289, Pls.' Mot.) According to Plaintiffs, this is the protocol authorized by the November 20, 2019 order. (Id.) Plaintiffs assert that the court should adopt their protocol for the use of TAR, which would require the City to use TAR on the entire ESI collection with an agreed-upon coding system for responsiveness. (Id.) In response, the City argues that the November 2019 order did not confine it to a particular methodology for identifying responsive or privileged ESI, and that TAR is an efficient and accurate tool for identifying both. (R. 300, Def.'s Resp.) The City also objects to Plaintiffs' proposed TAR protocol, arguing that the federal rules governing discovery impose no obligation on the responding party to conduct its responsiveness review in a manner dictated by the requesting party. (Id.)
A. Active Learning
As an initial matter, the court finds it necessary to clarify the type of TAR at issue and explain its key features. The City seeks to use Relativity's Active Learning ("AL"), a type of TAR software that uses learning algorithms to prioritize documents for its attorneys to review manually. (R. 300, Def.'s Resp. at 5-6.) As the City describes it:
[i]n AL review, like a manual review, search parameters are used to cull down a collected data set to a review set. That review set is then put into the AL application where the algorithms use data points collected through attorney review of documents in order to reorganize
the documents in the review queue in a more efficient order. With each coding decision the attorneys make, the technology continues to learn and prioritize which documents contain contextually similar content as documents which are coded as responsive. AL re-prioritizes the documents in the review queue every 20 minutes. The AL tool does not make any coding decisions about a document's responsiveness, privilege, confidentiality, or issue. It merely shuffles the order of the documents being reviewed based on the coding decisions [i.e., responsive or nonresponsive] made by the attorney review team. All documents marked responsive and ultimately produced are done so by human reviewers.(Id. at 6-7 (emphasis in original) (footnote and internal citations omitted).)
According to Plaintiffs, TAR software such as Relativity's AL allows parties to set aside and never review large portions of an ESI collection. (R. 289, Pls.' Mot. at 7.) There is some truth to this assertion because there comes a point when, based on the reviewers' coding decisions, the software establishes that the remaining documents in the queue are likely to be nonresponsive. It is then incumbent upon the reviewer to conduct sampling and other quality control tests to ensure that the remaining unreviewed documents are indeed irrelevant. The reviewer may of course forge ahead with his or her review, but typically documents identified as nonresponsive are neither reviewed nor produced. In short, the reviewer has discretion to decide when no further manual review is necessary. (See generally R. 300-2, Relativity's Assisted Review Active Learning Guide (June 8, 2020).)
The City proposes to use AL "to assist its attorneys with its responsiveness review to avoid the burden of conducting a manual attorney review" of the approximately 190,000 emails, or 1.3 million pages of documents, that hit upon Plaintiffs' search terms. (R. 300, Def.'s Resp. at 4.) As the City describes it, it intends to review only documents that meet a particular standard of relevance as determined by AL, and to discount documents falling below that standard. (Id. at 7-8.) The City also intends to use AL's quality control applications (such as Elusion testing), graphing results, family reconciliation, and a "cut off score," to ensure that an attorney reviews all potentially responsive documents. (Id. at 8-9.) Significantly, the parties agree that generally TAR is a far more accurate means of producing responsive ESI than manual review or keyword searches. (R. 289, Pls.' Mot. at 7; R. 300, Def.'s Resp. at 5.)
B. November 20, 2019 Order
Turning to the merits of the motion, Plaintiffs assert that the City's proposed use of AL is inconsistent with the court's November 20, 2019 order. (R. 289, Pls.' Mot. at 5.) In particular, Plaintiffs argue that because the parties have "always agreed that they would identify responsive emails" through keyword searches and the City never mentioned using TAR, under the November 2019 order, the City must use agreed-upon search terms to further reduce the ESI collection and then produce all of the nonprivileged documents that hit upon the search terms—regardless of whether they are responsive. (Id. at 9-10 (emphasis in original).) The City responds that "[n]othing in the [court's November 2019] [o]rder limits how the City may conduct its ESI responsiveness or privilege review or requires the City to negotiate with Plaintiffs concerning its review method." (R. 300, Def.'s Resp. at 4) (emphasis in original).)
The court agrees with the City that the November 2019 order did not set forth the review methodology that the City must use to identify responsive ESI. The order resolved issues regarding the method to be used for collecting and identifying the initial universe of emails. While the court anticipated that the parties would need to perform multiple keyword searches in order to narrow the universe of emails, it never directed them to do so. Nor did it suggest that after the searches are performed the City would have to produce the entire batch of documents subject only to a privilege review. In fact, the court specifically rejected Plaintiffs' proposal that the City produce all of the documents that hit upon their initial search terms without further review. While the City may dump all 1.3 million pages of documents on Plaintiffs with an entry of a Rule 502(d) order, it also has the right to perform a review to produce only those documents that are responsive and relevant. In sum, the City's responsiveness review is outside the scope of the November 2019 order.
C. Review Methodology
Aside from the November 2019 order, Plaintiffs point to no binding legal authority to support their request to force the City to use refined keyword searches to identify responsive ESI. They instead make a series of claims in an apparent effort to demonstrate that TAR is not appropriate for this case. Plaintiffs claim, for example, that TAR is a culling tool rather than a method of responsiveness review, (R. 289, Pls.' Mot. at 7), but this argument has no merit given Plaintiffs' own description of the software as a tool to "predict and apply responsiveness determinations," (id. at 6).
Plaintiffs also argue that because TAR is more effective at identifying responsive documents than traditional manual review, pre-TAR culling will eliminate large amounts of potentially relevant ESI. (Id. at 7.) The problem with this argument is that it assumes that those emails removed by the keyword searches likely would have been identified using TAR at the outset instead. Indeed, the low richness of the ESI collection in this case suggests just the opposite. The City's vendor collected over nine million pages of documents, less than 15% of which hit on Plaintiffs' own search terms. (See R. 300, Def.'s Resp. at 4.) While the court does not discount the possibility that using TAR at the onset might reveal more responsive documents overall, based on the number of documents that were discarded using Plaintiffs' proposed search terms, pre-TAR culling will achieve the best possible review in this case. In other words, it satisfies the reasonable inquiry standard and is proportional to the needs of this case under the federal rules. See generally Fed. R. Civ. P. 26.
Finally, Plaintiffs express concern that the attorney reviewers will improperly train the TAR tool by making incorrect responsiveness determinations or prematurely ending the review. (Id. at 7-8.) But these concerns are present no matter which methodology is employed. In traditional manual review for example, reviewers may have different interpretations of whether a particular document is responsive. Even a single reviewer may make a different relevancy determination based on his or her knowledge about the case at the time of the determination. In short, uncertainty in determining responsiveness is not unique to TAR. In any event, AL has a variety of quality control applications that the City intends to employ with its review. (See R. 300, Def.'s Resp. at 8-9.) Those applications negate Plaintiffs' concerns.
In the absence of any compelling argument from Plaintiffs, the court agrees with the City that as the responding party it is best situated to decide how to search for and produce emails responsive to Plaintiffs' discovery requests. (R. 300, Def.'s Mem. at 13) (citing, inter alia, The Sedona Principles, Third Edition, 19 SEDONA CONF. J. 1, Principle 6 ("Responding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own [ESI].")).) The City has disclosed the TAR software—Relativity's AL—it intends to use and how it intends to validate the review results, which in this case is sufficient information to make the production transparent. Plaintiffs' insistence that the City must collaborate with them to establish a review protocol and validation process has no foothold in the federal rules governing discovery. Moreover, using TAR on the entire ESI collection—when, as Plaintiffs aptly point out, the parties spent nearly a year litigating the protocol for collecting and searching the City's ESI—would be wasteful and unduly burdensome, and would further delay the resolution of this almost four-year-old case. For these reasons, the court declines to adopt Plaintiffs' alternate TAR protocol.
Conclusion
For the foregoing reasons, Plaintiffs' motion to compel is denied.
ENTER:
/s/ _________
Young B. Kim
United States Magistrate Judge