Opinion
No. 21 C 2046
2023-07-14
DEAL GENIUS, LLC, a Delaware Limited Liability Company, Plaintiff, v. O2COOL, LLC, a Delaware Limited Liability Company, Defendant.
David J. Thomas, Pro Hac Vice, Honigman LLP, Bloomfield Hills, MI, William Butler Berndt, Ron N. Sklar, Honigman LLP, Chicago, IL, for Plaintiff. Michael Anthony Parks, Sartouk H. Moussavi, Thompson Coburn LLP, Chicago, IL, for Defendant.
David J. Thomas, Pro Hac Vice, Honigman LLP, Bloomfield Hills, MI, William Butler Berndt, Ron N. Sklar, Honigman LLP, Chicago, IL, for Plaintiff. Michael Anthony Parks, Sartouk H. Moussavi, Thompson Coburn LLP, Chicago, IL, for Defendant.
ORDER ADDRESSING THE PARTIES' DISPUTE RE ELUSION TESTING AND ADDITIONAL SEARCH TERMS
Philip J. Favro, Special Master
The Special Master hereby issues the following ORDER pursuant to the Order Appointing Philip Favro as Special Master ("Special Master Appointment Order") and the Stipulation and Order Regarding Production of Responsive Emails ("1/30/23 Order") that empowers the Special Master to "issue an order deciding the issue(s) in dispute" between Plaintiff and Counterclaim-Defendant Deal Genius, LLC ("Deal Genius") and Defendant and Counterclaimant O2COOL, LLC ("O2COOL") over "whether and what additional relevant documents should be produced" from the Null Set after Deal Genius concludes elusion testing.
Docket No. 75, filed September 23, 2022.
Docket No. 109, at ¶7(d), filed January 30, 2023.
I. FACTUAL BACKGROUND
In this patent litigation involving claims and counterclaims for invalidity and infringement, the Parties have been "bitterly divided" for some time now over issues relating to the production of relevant emails. Those disputes involve, among other things, concerns from O2COOL that Deal Genius has failed to make fulsome email productions from its five designated email custodians in response to O2COOL's five email search queries. The five Deal Genius email custodians are Chris Matsakis, Monica Matsakis, Emily Greenfield, Dave Sienna, and Ram Subrahmanya. Those disputes eventually culminated in the appointment of the Special Master.
Docket No. 71, filed September 21, 2022.
Id.
In an effort to resolve those disputes, the Special Master worked with the Parties to execute a stipulation and order, which culminated in the 1/30/23 Order. The 1/30/23 Order required Deal Genius to redo its production of documents in response to certain of O2COOL's search queries ("Search Queries 2-5"). On March 3, 2023, Deal Genius produced 54 documents to O2COOL in response to the 1/30/23 Order.
Docket No. 109, filed January 30, 2023. The Parties executed a separate stipulation regarding Search Query One, which was entered as an Order on November 7, 2022. The production that Deal Genius made pursuant to the November 7, 2022 Order is not at issue and is not addressed in the present Order.
O2COOL subsequently raised multiple issues with Deal Genius's March 3, 2023 production. The only issue pending at this time before the Special Master arises from the requirement under the 1/30/23 Order that Deal Genius conduct elusion testing on the documents that did not hit on Search Queries 2-5 to ensure that relevant documents contained in the null set were not left unproduced to O2COOL.
O2COOL filed a request for clarification with the Court on May 12, 2023 seeking certain relief from the Court's May 11, 2023 Minute Order concerning the requirement that Deal Genius produce a list of duplicate documents pursuant to the 1/30/23 Order. While O2COOL's request for relief arises from the 1/30/23 Order, it is unrelated to the issue of elusion testing and is directed to the Court and not to the Special Master. The Special Master accordingly will not address that issue in the instant Order.
A. Elusion Testing Requirement from the 1/30/23 Order
The 1/30/23 Order required that Deal Genius—after making its production of relevant emails identified in response to Search Queries 2-5—conduct elusion testing by reviewing a sample of documents from the null set, i.e., the subset of documents that were either coded nonresponsive or that did not hit on Search Queries 2-5. If Deal Genius did identify relevant documents among the null set sample, the 1/30/23 Order required Deal Genius—after making its production of relevant emails identified in response to Search Queries 2-5—to produce those documents to O2COOL within 10 days of its previous production of documents. In addition, the 1/30/23 Order directed Deal Genius to disclose to O2COOL during that same 10-day period the number of documents in the null set, the null set sample document size, and the precise number of documents produced from the Null Set Sample.
The Sedona Conference defines the term "elusion" as "[t]he percentage of documents of a search's null set that were missed by the search, usually determined with review of a random sample of the null set. The elusion rate can be multiplied by the number of documents in the null set to estimate how many documents were missed by the search." The Sedona Conference Glossary: eDiscovery & Digital Information Management, Fifth Edition, 21 SEDONA CONF. J. 263, 304 (2020). The Sedona Conference respectively defines "null set" and "null set sample" as "[a] set of files that are not positive results of a search" and "[s]ampling a null set to search for false negatives of the search that created the null set." Id. at 344. The Sedona Conference is the preeminent legal institution dedicated to advancing thoughtful reforms on important legal issues. The organization is best known for its work on electronic discovery. The authoritative resources it prepares are regularly cited by courts both in the Northern District of Illinois and beyond as instructive on issues relating to electronic discovery. See, e.g., Gross v. Chapman, No. 19 C 2743, 2020 WL 4336062, at *2 (N.D. Ill. July 28, 2020); City of Rockford v. Mallinckrodt ARD Inc., 326 F.R.D. 489, 492, n.2, 493, 495 (N.D. Ill. 2018); Murray v. LeBlanc, No. CV 21-592-JWD-RLB, 2023 WL 2028927 (M.D. La. Feb. 15, 2023) (" 'District courts within the Fifth Circuit have acknowledged that the Sedona Principles and related Sedona commentaries are the leading authorities on electronic document retrieval and production.' "); Weidman v. Ford Motor Co., No. 2:18-CV-12719, 2021 WL 363560 (E.D. Mich. Feb. 3, 2021) ("The Sixth Circuit and this Court have recognized that the Sedona Conference's resources provide persuasive authority regarding discovery best practices and principles.").
The 1/30/23 Order specified that once Deal Genius completed its elusion testing obligations, the Parties then had seven days to meet and confer in order to determine whether Deal Genius should run additional search queries to identify other relevant documents that the search terms may not have identified. This effectively placed the burden on O2COOL to raise any issues regarding elusion testing with Deal Genius "within seven days following the production of any relevant documents found in the Null Set Sample."
Docket No. 109, at ¶7(c), filed January 30, 2023.
B. Deal Genius Produces Two Documents After Conducting Elusion Testing
On March 13, 2023, 10 days after Deal Genius produced 54 documents to O2COOL, Deal Genius disclosed to O2COOL that it performed elusion testing, its null set included 590,706 documents, its null set sample contained 2,397 documents, and it had identified two additional responsive documents from the null set sample that it was simultaneously producing to O2COOL. Under the terms of the 1/30/23 Order, the Parties had until March 20, 2023 to meet and confer "to determine if modifications to the search queries for the Terms 2-5 Emails are needed."
Id.
C. O2COOL's Meet and Confer Efforts with Deal Genius
After Deal Genius made its production from the null set sample, the Parties were silent regarding the issue of elusion testing until a conference with the Special Master on April 11, 2023 ("4/11/23 Conference"). During the 4/11/23 Conference, the Special Master reviewed with the Parties all active and known discovery issues and disputes. At the close of the 4/11/23 Conference and in response to the Special Master's query regarding the existence of any other out-standing discovery issues, the Parties indicated they were unaware of any further issues that needed to be addressed.
4/11/23 Conference, Transcript at 69:9-19.
The Special Master then questioned whether the Parties had met and conferred regarding any issues with elusion testing. Having determined that O2COOL had not raised any issues to date with Deal Genius regarding elusion testing, the Special Master instructed O2COOL that if there were any issues that needed to be addressed, it do so "in the near future."
4/11/23 Conference, Transcript at 72:1-3.
O2COOL subsequently contacted Deal Genius and requested that it run one additional search term to possibly ferret out other relevant documents that might be in the null set and which the search terms had not yet identified. After exchanges of meet and confer correspondence and in preparation for a discovery conference with the Special Master on May 2, 2023, Deal Genius agreed to run the additional search query that O2COOL proposed—neck! & fan & (karnani OR nagrani)—against the null set of documents ("First Modified Search Term"). The First Modified Search Term, a modified version of Search Query 2 (neck! & fan & infring!), yielded 28 hit documents, all of which Deal Genius deemed relevant upon review. Deal Genius subsequently produced these 28 documents to O2COOL on May 12, 2023.
See Correspondence from Ron N. Sklar to the Special Master, dated May 1, 2023. Deal Genius disputed the need to run the First Modified Search Term, but agreed to do so "to eliminate disputes and to proceed with the merits of the case." Id. at 2.
O2COOL thereafter requested that Deal Genius run one additional search term: fan & (karnani or nagrani) ("Second Modified Search Term"). O2COOL initially argued that running the Second Modified Search Term was merited given that the First Modified Search Term (neck! & fan & (karnani OR nagrani)) yielded only 28 hit documents and that the Parties were obligated under the 1/30/23 Order to work cooperatively and transparently to find relevant documents that Search Queries 2-5 did not identify.
In response, Deal Genius refused to undertake any further searches of the null set documents. In summary, Deal Genius asserts that the Second Modified Search Term would not target information even remotely connected to the two documents it produced from the null set sample and that such information (even assuming it was relevant) could have been obtained during fact discovery, which closed long ago. Deal Genius also argues that O2COOL's request was untimely under the express provisions of the 1/30/23 Order, which required O2COOL to raise issues arising from the elusion testing within seven days (March 20, 2023) following Deal Genius's March 13, 2023 production of two documents after reviewing the null set sample.
For its part, O2COOL maintains that the Second Modified Search Term targets information directly related to the two responsive documents Deal Genius produced from the null set sample. In addition, O2COOL argues that it could not have obtained such information during fact discovery given that Deal Genius allegedly provided inaccurate hit count information to O2COOL. Finally, O2COOL represented that the Second Modified Search Term is its "final" follow up request for information during the elusion testing process and that its request was not time barred under the 1/30/23 Order.
The Parties have sharply contended over the issue of hit count information for many months now. O2COOL asserts that Deal Genius provided "false hit counts" during the Parties' meet and confer over O2COOL's search terms, an accusation which Deal Genius vehemently denies. This issue is now a subject of O2COOL's pending motion for sanctions against Deal Genius. See Docket No. 139, at ¶7, filed June 20, 2023.
D. O2COOL's Search Term Yields 50 Additional Hit Documents
With the Parties now at an impasse, 1/30/23 Order requires that the Special Master issue an order deciding the Parties' elusion testing dispute over the Second Modified Search Term. To help clarify the issues in dispute and to appropriately weigh proportionality considerations, the Special Master directed Deal Genius on June 2, 2023 to run the Second Modified Search Term and share the total number of hits (family complete), the unique hits, and the overall size of the null set with O2COOL and the Special Master. Deal Genius readily complied with the Special Master's directive and on June 5, 2023, revealed that the Second Modified Search Term yielded 50 hit documents (family complete), 18 of which were unique documents. Deal Genius also revealed that the size of its null set had now grown to 662,502 documents.
See Correspondence from Ron N. Sklar to the Special Master, dated June 5, 2023. Deal Genius initially revealed that the null set included 590,800 documents. See Correspondence from Ron N. Sklar to Sartook H. Moussavi, Michael A. Parks, and Bradley H. Thomas, dated December 19, 2022. Deal Genius next indicated that the null set encompassed 590,706 documents. See Correspondence from Ron N. Sklar to Sartook H. Moussavi and Michael A. Parks, dated March 13, 2023. Deal Genius does not explain why its null set figures have varied.
Afterwards and in response to further questioning from the Special Master, the Parties indicated that—despite the results from the Second Modified Search Term hit report—their positions remained unchanged. To wit, Deal Genius refused to review the documents identified by the Second Modified Search Term while O2COOL insisted that Deal Genius must do so and produce any relevant documents. Accordingly, the Special Master has prepared the instant Order resolving this dispute.
II. DISCUSSION
A. Legal Standards
1. Relevance and Proportionality
The scope of discovery in federal court is broad. Parties may obtain information that is relevant to the claims and defenses at issue. Relevance—and not admissibility—is the initial touchstone, with Rule 26(b)(1) providing for such discovery even if it is ultimately not admissible as evidence.
See Gamon Plus, Inc. v. Campbell Soup Co., No. 1:15-CV-08940, 2022 WL 18284320, at *1 (N.D. Ill. May 26, 2022); Duracell U.S. Operations, Inc. v. JRS Ventures, Inc., No. 17 C 3166, 2018 WL 704686, at *7 (N.D. Ill. Feb. 5, 2018).
Id.; see Duracell, 2018 WL 704686, at *7.
And yet, discovery is not without bounds. Since December 1, 2015, the scope of discovery is expressly conditioned by a second touchstone, proportionality. As a result, parties may only obtain relevant information if they establish that it is "proportional to the needs of the case." Rule 26(b)(1) specifically identifies six proportionality factors, which are as follows: "[T]he importance of the issues at stake in the action, the amount in controversy, the parties' relative access to relevant information, the parties' resources, the importance of the discovery in resolving the issues, and whether the burden or expense of the proposed discovery outweighs its likely benefit."
See Saleh v. Pfister, No. 18 C 1812, 2021 WL 326361, at *2 (N.D. Ill. Feb. 1, 2021).
FED. R. CIV. P. 26(b)(1); see United States ex rel. Gill v. CVS Health Corp., No. 18 C 6494, 2023 WL 4106267, at *1 (N.D. Ill. June 20, 2023); Philip Favro, Proportionality Weaponized: How It Happens and What Can Parties and Courts Do About It, THE CIRCUIT RIDER: THE JOURNAL OF THE SEVENTH CIRCUIT BAR ASSOCIATION (Sept. 24, 2021) ("Proportionality Weaponized").
Id.
In principle, none of the proportionality factors should necessarily predominate and courts should apply them in an evenhanded manner and determine which have greater application in a particular matter. In practice, however, the final two factors—the importance of the requested discovery and balancing discovery benefits against its burdens—are generally the most important and determinative of the issues. Under the subjective balancing test created by those factors, requesting parties should identify the benefits of the sought-after discovery and why they ultimately need it. Responding parties must in turn supply hard information substantiating claimed burdens.
See FED. R. CIV. P. 26(b)(1), 2015 committee note ("The court's responsibility, using all the information provided by the parties, is to consider these and all the other factors in reaching a case-specific determination of the appropriate scope of discovery."); United States Equal Emp. Opportunity Comm'n v. George Washington Univ., No. 17-CV-1978 (CKK/GMH), 2020 WL 3489478, at *4 (D.D.C. June 26, 2020) (" '[N]o single factor is designed to outweigh the other factors in determining whether the discovery sought is proportional.' ").
See George Washington Univ., 2020 WL 3489478, at *12 ("Engaging in the case-by-case analysis that is required when analyzing the proportionality of requested discovery . . . it is clear that the burden imposed on GW to comply with these discovery requests as written outweighs the likely benefit."); Oxbow Carbon & Mins. LLC v. Union Pac. R.R. Co., 322 F.R.D. 1, 6 (D.D.C. 2017) ("To be sure, however, 'the [2015] amendments to Rule 26(b) do not alter the basic allocation of the burden on the party resisting discovery to—in order to successfully resist a motion to compel—specifically object and show that . . . a discovery request would impose an undue burden or expense or is otherwise objectionable.' "); Favro, Proportionality Weaponized, at 14-15.
Id.
Id.; The Sedona Conference, Commentary on Proportionality in Electronic Discovery, 18 SEDONA CONF. J. 141, 167 (2017) ("Burden and expense should be supported by hard information and not by unsupported assertions.")
2. Search Terms
When responding to Rule 34 production demands and conducting a search through massive volumes of electronically stored information ("ESI"), parties typically rely on electronic search methodologies to help identify relevant information. Search terms are a commonly used method that lawyers and their clients use to identify relevant documents. While search terms can be an effective approach for targeting relevant information, courts have acknowledged several known limitations with search terms that can reduce their effectiveness if not properly addressed by the parties.
See, e.g., Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 256-57 (D. Md. 2008) ("while it is universally acknowledged that keyword searches are useful tools for search and retrieval of ESI, all keyword searches are not created equal; and there is a growing body of literature that highlights the risks associated with conducting an unreliable or inadequate keyword search.")
One of the most problematic limitations of search terms is that they tend to be under-inclusive. Human language and communications are complex. Because search terms only hit on documents with the exact word or set of words encompassed by the terms at issue—including wildcards (*) and expanders (!)—relevant documents that do not include the precise wording reflected in a search term or phrase will be excluded from the search results. Also referred to as "false negatives," under-inclusive search results can leave key discoverable information undetected and thus not produced.
See Shumway v. Wright, No. 4:19-CV-00058-DN-PK, 2019 WL 8137128, at *3 (D. Utah Nov. 25, 2019), report and recommendation adopted, 2020 WL 1042211 (D. Utah Jan. 8, 2020) ("search terms have several known limitations (including being both under-inclusive and over-inclusive) that often reduce their effectiveness.").
Id.
See City of Rockford v. Mallinckrodt ARD Inc., 326 F.R.D. 489, 493 (N.D. Ill. 2018) ("With key word searching (as with any retrieval process), without doubt, relevant documents will be produced, and without doubt, some relevant documents will be missed and not produced.")
To ameliorate the impact of under-inclusive search results (and other limitations with search terms), parties should adopt iterative search term development procedures and quality assurance practices that can help identify relevant information not retrieved by the original search terms. One of the most important practices that a producing party can implement at the conclusion of the process is elusion testing.
William A. Gross Construction, Inc. v. American Manufacturers Mutual Insurance Company, 256 F.R.D. 134, 136 (S.D.N.Y. 2009) ("where counsel are using keyword searches for retrieval of ESI, they at a minimum must carefully craft the appropriate keywords . . . and the proposed methodology must be quality control tested to assure accuracy in retrieval.").
See City of Rockford, 326 F.R.D. at 493-94.
3. Elusion Testing
Elusion testing is a standard quality assurance practice in electronic discovery and essential for providing transparency on whether material documents have eluded the search terms. Elusion testing is generally considered a "validation" measure because it can ultimately validate, or confirm the reasonableness and proportionality, of the producing party's production and thus help satisfy the Rule 26(g) reasonable inquiry standard.
See id. at 494-95 (N.D. Ill. 2018) (discussing the purpose of elusion testing and the need for reviewing the null set sample to ensure a production is proportional); see also Shumway v. Wright, No. 4:19-cv-00058, 2019 WL 8135308 (D. Utah Nov. 22, 2019), report and recommendation adopted, 2019 WL 8137130 (D. Utah Dec. 16, 2019).
See City of Rockford, 326 F.R.D. at 494.
Elusion testing involves having the producing party review a random sample from the "null set," which comprises the documents that did not hit on any of the search terms, together with the documents the producing party deemed not responsive after reviewing the search term hits. The producing party reviews the documents in null set sample, determines whether there are any relevant documents, and, to the extent there are, produces those relevant documents to the requesting party.
See id.; Shumway, 2019 WL 8135308 at *1. The producing party typically prepares a random sample of documents from the null set, which is calculated using a statistical confidence level of 95% with a margin of error of 2%. Depending on the final size of the null set, the null set sample typically will contain between 1,000 and 2,400 documents.
See id.
If relevant documents are indeed identified and produced, the parties should next meet and confer to assess the quality and nature of the produced documents from the null set and then determine whether the producing party should conduct additional searches to identify any additional relevant information. The parties could undertake a purely quantitative statistical analysis and calculate the elusion rate by dividing the number of relevant documents identified in the null set sample against the total null set sample size. If the "elusion rate" falls below a certain benchmark on which the parties have agreed (e.g., two percent), the parties may decide to forego any further searches for relevant information in the null set. However, where the overall null set is large and the percentage of relevant documents in the overall document population is low to begin with (commonly referred to among eDiscovery practitioners as "low richness"), the parties may consider running additional search terms against the null set despite a low elusion rate. This approach to validation of a search and retrieval process can better safeguard against the possibility that discoverable information remains concealed in the null set due to the sheer volume of data collected.
See Maura R. Grossman and Gordon V. Cormack, The Grossman-Cormack Glossary of Technology-Assisted Review, 7 FED. CTS. L. REV. 1, 15 (2013) ("Glossary of Technology-Assisted Review").
Id.
B. Analysis
Applying the above-referenced standards to the instant discovery dispute, and after examining the Parties' meet and confer correspondence and reviewing their submissions, the Special Master finds that O2COOL's request that Deal Genius produce all relevant documents identified in response to the Second Modified Search Term is both reasonable and proportional under the circumstances. There are several reasons that support the Special Master's finding.
1. Relevance
There does not appear to be any dispute between the Parties regarding the relevance of the documents that O2COOL seeks by virtue of the Second Modified Search Term. Instead, the Parties dispute whether the Second Modified Search Term is sufficiently connected to the documents Deal Genius produced after its null set sample review. While Deal Genius disputes O2COOL's contention of a causal link between the produced null set documents and the Second Modified Search Term, nowhere in Deal Genius's meet and confer correspondence or submissions to the Special Master is there any contention that the documents sought by the Second Modified Search Term are not relevant.
See, e.g., Correspondence from Ron N. Sklar to Sartouk H. Moussavi, dated May 17, 2023; Correspondence from Ron N. Sklar to the Special Master, dated May 26, 2023.
The 1/30/23 Order does not require a causal connection to the relevant documents located in the original null testing sample. Instead, the 1/30/23 Order makes clear that the purpose for reviewing the null set sample is "to ensure that Deal Genius produces other relevant documents (assuming there are any additional relevant documents) from the Null Set that are material to the claims and defenses of the above-captioned litigation."
Docket No. 109, at ¶7(c).
Moreover, O2COOL has repeatedly emphasized the relevance of the information it seeks by way of the Second Modified Search Term. In particular, O2COOL maintains that the documents the Second Modified Search Term targets are relevant to its willful infringement claim. Given the foregoing, the Special Master concludes that the Second Modified Search Term is designed to obtain relevant information.
See, e.g., Correspondence from Sartouk H. Moussavi to Ron N. Sklar, dated May 11, 2023 and May 18, 2023.
See Correspondence from Sartouk H. Moussavi to Ron N. Sklar, dated May 11, 2023.
2. Proportionality
The Special Master has not found any arguments in the Parties' meet and conference correspondence or their submissions that expressly apply the Rule 26(b)(1) proportionality factors to the instant dispute. Nevertheless, the Special Master is duty-bound by the Federal Rules of Civil Procedure to address proportionality considerations and determine whether proportionality warrants or militates against authorizing the requested discovery.
See FED. R. CIV. P. 26(b)(1); Victor Stanley, Inc. v. Creative Pipe, Inc., 269 F.R.D. 497, 523 (D. Md. 2010) ("courts have tended to overlook the importance of proportionality in determining whether a party has complied with its duty to preserve evidence in a particular case . . . all permissible discovery must be measured against the yardstick of proportionality.").
Viewing O2COOL's request through this lens, it is readily apparent that the discovery O2COOL seeks is proportional to the needs of this case. Under the critical fifth proportionality factor, O2COOL has advanced a compelling reason to substantiate the importance of the requested discovery, i.e., the modified search term is designed to "target the most pertinent emails relating to O2COOL's willful infringement theory." In contrast, Deal Genius has not offered any evidence or argument pursuant to the sixth proportionality factor that would suggest it would be unduly burdensome to review the 50 documents (family complete), only 18 of which are unique hits, stemming from the Second Modified Search Term. Indeed, it would strain credulity for a party to argue that it would be unduly burdensome to review such a limited number of documents. Instead, Deal Genius relies on two procedural arguments to stave off the requested discovery, neither of which can withstand scrutiny.
See Correspondence from Sartouk H. Moussavi to Ron N. Sklar, dated May 11, 2023.
3. The Second Modified Search Term is Not Time-Barred
First, Deal Genius argues that O2COOL is time-barred from running the Second Modified Search Term because O2COOL failed to raise any elusion testing issues within the 1/30/23 Order's seven-day meet and confer period. The Special Master concludes that Deal Genius is correct that O2COOL did not raise elusion testing issues within the Court-ordered period of seven days. Indeed, O2COOL waited nearly a month to do so and only did so after the Special Master unilaterally raised the question of elusion testing with the Parties during the 4/11/23 Conference.
Nevertheless, Deal Genius did not argue that O2COOL's First Modified Search Term request was untimely. Instead, Deal Genius agreed to run the First Modified Search Term, which eventually identified 28 documents, all of which Deal Genius produced to O2COOL. That Deal Genius willingly ran the First Modified Search Term operates as a waiver of any right Deal Genius might have been able to assert under the 1/30/23 Order against running the Second Modified Search Term. Even without the waiver, given the general judicial preference to resolve disputes on the merits rather than on procedural issues, the Special Master would be reluctant to strictly enforce the seven-day deadline and possibly deprive O2COOL of relevant information for its claims. For both of these reasons, the Special Master rejects Deal Genius's position that O2COOL is time-barred from pursuing discovery of the 18 unique documents (50 documents family completed) retrieved by the Second Modified Search Term.
See Rubman v. U.S. Citizenship & Immigr. Servs., 800 F.3d 381, 392 (7th Cir. 2015) ("A waiver is ordinarily an intentional relinquishment or abandonment of a known right or privilege.").
See LKQ Corp. v. Gen. Motors Co., No. 20 C 02753, 2022 WL 14634800 (N.D. Ill. Oct. 25, 2022); DR Distributors, LLC v. 21 Century Smoking, Inc., 513 F. Supp. 3d 839, 983 (N.D. Ill. 2021) ("But public policy favors decisions being made on the merits.").
4. O2COOL is Within Its Rights to Obtain the Requested Discovery
Second, Deal Genius argues that the Second Modified Search Term is inappropriate because it targets discovery that O2COOL could have obtained before the end of fact discovery in the instant litigation. This is not the first time that Deal Genius has sparred with O2COOL over this issue, to wit, whether O2COOL may obtain discovery of information after the close of fact discovery relating to nonparties Deepak Nagrani and Kumar Karnani.
For purposes of this Order, it should be noted that the Court—pursuant to the Special Master's recommendation—previously denied without prejudice O2COOL's request to obtain emails from Deal Genius regarding Messrs. Nagrani and Karnani or leave to take their depositions. O2COOL's request—lodged in the context of trying to identify supplemental or replacement evidence for the lost Emily Greenfield emails—was denied because the Court provided O2COOL with a more effective remedy to address this issue, i.e., authorizing the deposition of Ms. Greenfield. Accordingly, the Special Master recommended and the Court agreed that discovery focused on Messrs. Karnani and Nagrani would not be appropriate at that time. The Special Master also observed in dicta that affirmative discovery on Messrs. Nagrani and Karnani would be improper because O2COOL likely had sufficient opportunity to previously obtain information from Messrs. Karnani and Nagrani before the close of fact discovery.
Docket No. 122, filed Mar. 21, 2023; Docket No. 114, filed Feb. 23, 2023; Docket No. 112, at ¶¶34-35, filed Feb. 17, 2023.
Docket No. 122, filed Mar. 21, 2023; Docket No. 112, at ¶¶34-35, filed Feb. 17, 2023.
Id.
Docket No. 112, at ¶36, filed Feb. 17, 2023.
To be certain, Deal Genius's argument on this point has some merit. The Special Master still has concerns that O2COOL is using the Second Modified Search Term to obtain discovery regarding Messrs. Karnani and Nagrani that it could have sought during fact discovery. Context matters, though, and in the context of the present dispute, those concerns and Deal Genius's present argument are both outweighed by the need to ensure that potentially key documents are produced in response to the Second Modified Search Term. This is particularly the case since O2COOL has long disputed the adequacy of the email searches that Deal Genius ran during fact discovery and maintains that relevant information involving Messrs. Karnani and Nagrani would have been produced during fact discovery had Deal Genius properly handled O2COOL's Search Queries. In addition, the Special Master previously concluded that Deal Genius's discovery efforts in response to Search Queries 1-5 were incomplete, which resulted in the production of well over two thousand additional documents to O2COOL after the close of fact discovery.
Id. at ¶28.
This Court has repeatedly emphasized that the application of proportionality standards requires a "common sense and experiential assessment." Given the well-chronicled problems Deal Genius has experienced conducting ESI searches and the efforts O2COOL has made to address the issues, common sense in this instance requires Deal Genius to undertake the modest burden of reviewing 50 additional documents and producing any relevant materials to O2COOL.
See, e.g., Velez v. City of Chicago, No. 18 C 8144, 2021 WL 1978364 at *5 (N.D. Ill. May 17, 2021).
5. Further Elusion Testing Is Warranted
The foregoing analysis of relevance and proportionality alone justifies granting O2COOL's request and ordering Deal Genius to produce any relevant information among the documents identified in response to the Second Modified Search Term. And yet, there is another compelling reason that warrants this conclusion: Concern about the elusion testing process that Deal Genius performed.
Deal Genius conducted its elusion testing obligation from the 1/30/23 Order by reviewing the null set sample (2,397 documents) and then producing the only two files it identified as relevant. Using a statistical analysis, the elusion rate—the number of relevant documents in the null set that eluded Deal Genius's review—is 0.08% (2 ÷ 2,397 = 0.00083 or 0.08%). If the elusion rate of 0.08% is applied to the full null set of 662,502 documents, a pinpoint estimate of the remaining relevant documents in the null set would be approximately 530. Randomly searching through over 660,000 documents to identify such a small number of ostensibly relevant documents would, on its face, seem disproportionate to the needs of this case. Yet, strikingly, the First Modified Search Term returned 28 unique documents, all of which Deal Genius deemed relevant and produced to O2COOL. Furthermore, the Second Modified Search Term retrieved 18 unique documents (with a total of 50 documents, family complete).
See Correspondence from Ron N. Sklar to Sartouk H. Moussavi and Michael A. Parks, dated March 13, 2023.
The outcome of the First Modified Search Term raises questions regarding the reliability of Deal Genius's elusion testing results. For example, Deal Genius implicitly acknowledges that its null set includes an excessive number of documents—potentially hundreds of thousands of emails—beyond the scope of discovery. Even though the precise volume remains unknown, the presence of so many irrelevant documents in the null set skews the ability of the null set sample to capture relevant documents that Search Queries 2-5 did not identify. This is evidenced by the results of the First Modified Search Term; Deal Genius's null set sample did not include any of the 28 hit documents—all of which turned out to be relevant—that it subsequently identified when it ran the First Modified Search Term. The 0.08% elusion rate may therefore be "misleading" and standing alone, is not determinative of whether Deal Genius should be obligated to conduct additional searches for relevant information.
The Court ordered Deal Genius to review approximately four years of email from the five email custodians. See Docket No. 109, at ¶2, filed January 30, 2023. Nevertheless, Deal Genius pulled the entire custodian email accounts for two of the five email custodians, Chris Matsakis and Monica Matsakis. See January 11, 2023 Conference Transcript, at 16:11-17:11 ("[Special Master] Why did Deal Genius export the entire mailboxes for Chris Matsakis and Monica Matsakis when they're over a hundred gigabytes respectively? . . . [Counsel for Deal Genius] our process was to export everything and then limit to date range after that . . . this way we have everything. There can be no dispute. We've pulled all of the custodial documents and put it into the Casepoint system."). The total number of emails that Deal Genius collected from Chris Matsakis (362,580) and Monica Matsakis (292,047) far exceed what Deal Genius ultimately collected from its other custodians, Dave Sienna (90,782) and Ram Subrahmanya (91,402), which were apparently limited to the relevant time period. See Correspondence from Ron N. Sklar to the Special Master, dated November 11, 2022.
Grossman and Cormack, Glossary of Technology-Assisted Review, 7 Fed. Cts. L. Rev. at 15 ("A low Elusion value has commonly been advanced as evidence of an effective search or review effort . . . but that can be misleading because it quantifies only those Relevant Documents that have been missed by the search or review effort; it does not quantify the Relevant Documents found by the search or review effort.") (emphasis in original).
Instead, the results of the First Modified Search Term demonstrably establish the need pursuant to the 1/30/23 Order for the Parties to consider running additional search terms to identify relevant information that may have eluded Search Queries 2-5. The Second Modified Search Term, which Deal Genius admits yielded only 18 unique documents, appears reasonably and proportionally calculated to obtain such information under Rule 26(b)(1). Given these circumstances, an order directing Deal Genius to (1) review the documents that hit on the Second Modified Search Term, and (2) produce any corresponding relevant materials (family complete) to O2COOL, is clearly within the scope and intent of the 1/30/23 Order and is warranted under the circumstances of this dispute.
See also Fed. R. Civ P. 26(g)(1), (3).
III. CONCLUSION
In conclusion, the Special Master hereby ORDERS as follows:
1. Deal Genius's objections to reviewing documents identified in response to the Second Modified Search Term are overruled;
2. Deal Genius shall review all documents identified in response to the Second Modified Search Term and produce any relevant documents to O2COOL on or before July 14, 2023;
3. Deal Genius shall produce any relevant documents identified in response to the Second Modified Search Term both family complete and in the production format on which the Parties previously agreed; and
4. Deal Genius shall serve a privilege log on or before July 21, 2023 that memorializes claims of privilege over any relevant documents among the documents identified in response to the Second Modified Search Query.