Summary
discussing the advantages of TAR but deferring to the parties' choice to use search terms
Summary of this case from In re Mercedes-Benz Emissions Litig.Opinion
Donald E Haviland, Jr, Jay William Chamberlin, William H. Platt, II, Ambler, PA, Peter J. Flowers, Jonathan Peter Mincieli, Michael W. Lenert, Meyers & Flowers, LLC, St. Charles, IL, Ifeanyichukwu C. Mogbana, Kerry Franklin Partridge, City Of Rockford Department Of Law, Rockford, IL, for Plaintiffs
George Patrick Watson, Lindsay Sklar Johnson, Bryan Cave LLP, Atlanta, GA, Herbert R. Giorgio, Jr., Rebecca A. Nelson, Bryan Cave Leighton Paisner LLP, St. Louis, MO, Philip Dean Bartz, Bryan Cave Leighton Paisner LLP, Washington, DC, Scott Collins Sullivan, Williams McCarthy LLP, Kerry Franklin Partridge, Ifeanyichukwu C. Mogbana, City of Rockford Department of Law, Joel M. L. Huotari, Rockford, IL, Jonathan Peter Mincieli, Michael W. Lenert, Peter J. Flowers, Meyers & Flowers, LLC, St. Charles, IL, Matthew M. Martino, Evan Ross Kreiner, Michael H. Menitove, Patrick G. Rideout, Robert A. Fumerton, James A. Keyte, Thomas Pak, Skadden, Arps, Slate, Meagher & Flom LLP, New York, NY, Amy Lynn Van Gelder, Eric J. Gorman, Matthew Robert Kipp, Skadden Arps Slate Meagher & Flom, LLP, David M. Hundley, Pendley, Baudin & Coffin, L.L.P., Chicago, IL, Adam Michael Foster, Michael L. Baum, Pedram Esfandiary, R. Brent Wisner, Baum Hedlund Aristei & Goldman, P.C., Jason D. Russell, Skadden Arps Slate Meagher and Flom LLP, Los Angeles, CA, Christopher L. Dueringer, Sarah Burwick, Bryan Cave LLP, Santa Monica, CA, Donald E. Haviland, Jr, William H. Platt, II, Jay William Chamberlin, Ambler, PA, for Defendants
ORDER ESTABLISHING PRODUCTION PROTOCOL FOR ELECTRONICALLY STORED INFORMATION
Iain D. Johnston, U.S. Magistrate Judge
"[A]s we know there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know..."
Donald Rumseld.
United States Department of Defense News Briefing (Feb. 12, 2002), http://archive.defense.gov/Transcripts/Transcript.aspx?TranscriptID=2636.
But what if there were a way to know the known unknowns? Would people use the process? In litigation, it turns out maybe - or maybe not. Sometimes, the answer is bad. Consequently, parties and attorneys would rather not know, particularly when a financial burden may incur just to learn something adverse to the party’s position, and, to make matters worse, may result in even further financial burdens. Additionally, many litigators (especially defense counsel) are pessimists, perhaps by nature, but certainly by training. They see danger everywhere. But, of course, it is also possible that the answer is good, and would be useful in the litigation. Counsel should not lose sight of that possibility.
A. BACKGROUND
These cases relate to the pricing of the prescription medication Acthar. Acthar is prescribed to treat, among other things, adults with multiple sclerosis and infants with infantile spasms, a rare seizure disorder. http://www.pdr.net/drug-summary/H-P--Acthar-corticotropin-1345 (last visited Aug. 7, 2018). They have been consolidated for purposes of discovery. In its case, the City of Rockford claims, among other things, that Defendant breached its contract, engaged in racketeering, and violated various antitrust statutes. Dkt. # 84. MSP makes similar claims in its case. Dkt. # 10. These cases likely involve millions of documents, the vast majority of which are stored electronically. These cases are also classic asymmetrical discovery cases: Mallinckrodt ARD Inc.’s production of documents will be astronomically larger than the City of Rockford’s production. Accordingly, the burdens - financial and others - in producing documents will disproportionately fall on Mallinckrodt.
B. ISSUE
Currently before the Court is the parties’ dispute as to a single aspect of the electronically stored information (ESI) production protocol. To date, the parties have generally worked cooperatively on discovery issues, including ESI issues, and have only raised legitimate discovery disputes that they were unable to resolve on their own. The attorneys are commended for this cooperation, and their clients should appreciate their efforts in this regard. The Court certainly does. The litigation so far is a solid example that zealous advocacy is not necessarily incompatible with cooperation. The current issue before the Court is an example of that advocacy and cooperation. The parties have worked to develop a protocol for the production of ESI in this case, but have now reached an impasse as to one aspect of the protocol.
In relevant part, here’s what the parties have agreed on. Initially, the parties will use key word searching of the ESI. If, at some point, a producing party seeks to use technology assisted review (TAR), also known as predictive coding, to identify or cull documents to be reviewed or produced, the producing party will notify the opposing party to discuss the protocol for that type of review. Moreover, as to the key word searching, the parties have agreed to the following search procedure.
• The parties will discuss and attempt to reach agreement on search methodologies, and anticipate agreeing on search terms, date restrictions, and custodian restrictions.
• Before implementing search terms, the producing party will provide a list of proposed search terms to the requesting party and the parties will discuss those terms and any additional terms proposed by the requesting party.
• Following the search, the producing party will provide a hit report, which will include the number of documents that hit on each term, the number of unique documents that hit on each term, and the total number of documents that would be returned by using the proposed search term list.
• If a party disputes a specific term as being overly broad, the producing party can agree to review a statistically valid sample of documents to determine if the term is returning mostly responsive documents. The producing party will then provide the report and any responsive documents retrieved to allow the parties to make any modifications as to the term in dispute.
• If the parties still disagree, then they will submit the dispute to the Court to resolve.
That is all reasonable and fulfills the goals of Federal Rule of Civil Procedure 1; namely, to achieve a just, speedy, and inexpensive determination of this case. This is the type of process courts routinely require, but see less often. Ferring B.V. v. Fera Pharms., LLC, No. CV 13-4640, 2016 WL 5396620, at *3-4, 2016 U.S. Dist. LEXIS 132522, at *9-10 (S.D.N.Y. Sept. 27, 2016).
Here’s where the parties disagree. What happens after the production? Defendants’ proposal is that if "the requesting party reasonably believes that certain categories of requested documents exist that were not included in the production, the parties will meet and confer to discuss whether additional terms are necessary." Dkt. # 153 (Case No. 17 CV 50107), Dkt. # 132 (Case No. 18 CV 379). In contrast, Plaintiffs propose a random sample of the null set. (The "null set" is the set of documents that are not returned as responsive by a search process, or that are identified as not relevant by a review process. See Maura R. Grossman & Gordon v. Cormack, The Grossman-Cormack Glossary of Technology-Assisted Review, 7 Fed. Cts. L.Rev. 1, 25 (2013). The null set can be used to determine "elusion," which is the fraction of documents identified as non-relevant by a search or review effort that are, in fact, relevant. Elusion is estimated by taking a random sample of the null set and determining how many or what portion of documents are actually relevant. Id. at 15.) Specifically, Plaintiffs propose the following provision: "The producing party agrees to quality check the data that does not hit on any terms (the Null Set) by selecting a statistically random sample of documents from the Null Set. The size of the statistically random sample shall be calculated using a confidence level of 95% and a margin of error of 2%. If responsive documents are found during the Null Set review, the producing party agrees to produce the responsive documents separate and apart from the regular production. The parties will then meet and confer to determine if any additional terms, or modifications to existing terms, are needed to ensure substantive, responsive documents are not missed." Dkt. # 153 (Case No. 17 CV 50107), Dkt. # 132 (Case No. 18 CV 379).
The Court pauses here for a moment to calm down litigators less familiar with ESI. (You know who you are.) In life, there are many things to be scared of, including, but not limited to, spiders, sharks, and clowns - definitely clowns, even Fizbo. ESI is not something to be scared of. The same is true for all the terms and jargon related to ESI. Discovery of ESI is still discovery, governed by the same Federal Rules of Civil Procedure as all other civil discovery. Brown v. Tellermate Holdings, Ltd., No. 11 CV 1122, 2014 WL 2987051, at *1-2, 2014 U.S. Dist. LEXIS 90123, at *4 (S.D. Ohio July 1, 2014) ("[T]he underlying principles governing discovery do not change just because ESI is involved."). So don’t freak out. Having said that, the ethical rules now require attorneys to be competent with technologies such as ESI. ABA Model Rule 1.1, Comment 8 ("To maintain the requisite knowledge and skill, a lawyer should keep abreast of changes in the law and its practice, including the benefits and risks associated with relevant technology..."). A good way for attorneys to increase their competency and comfort level with ESI is to review the educational materials available from the Seventh Circuit’s Electronic Discovery Pilot Program. See www.discoverypilot.com . Attorneys can also familiarize themselves with ESI terms by referring to the Grossman-Cormack Glossary of Technology-Assisted Review. 7 Fed. Cts. L.Rev. 1 (2013); see also The Sedona Conference Glossary: E-Discovery & Digital Information Management, 15 Sedona Conf. J. 305 (2014). The Grossman-Cormack Glossary of Technology-Assisted Review is to ESI as Stedman’s Medical Dictionary is to personal injury and Social Security disability litigation.
C. ANALYSIS
Federal Rule of Civil Procedure 26(g) requires all discovery requests be signed by at least one attorney (or party, if proceeding pro se ). Fed.R.Civ.P. 26(g)(1). By signing the response, the attorney is certifying that to the best of counsel’s knowledge, information, and belief formed after a reasonable inquiry, the disclosure is complete and correct at the time it was made. Fed.R.Civ.P. 26(g)(1)(A). But disclosure of documents need not be perfect. Winfield v. New York, 15 CV 5236, 2017 WL 5664852, at *9, *11, 2017 U.S. Dist. LEXIS 194413, at *29, *34 (S.D.N.Y. Nov. 27, 2017); Pension Committee of the University of Montreal Pension Plan v. Banc of America Securities, LLC, 685 F.Supp.2d 456, 461 (S.D.N.Y. Jan. 15, 2010) (overruled, in part, on other grounds) ("In an era where vast amounts of electronic information is available for review, discovery in certain cases has become increasingly complex and expensive. Courts cannot and do not expect that any party can meet a standard of perfection."); Federal Housing Finance Agency v. HSBC North America Holdings, Inc., 2014 WL 584300, at *2-3, 2014 U.S. Dist. LEXIS 19156, at *32 (S.D.N.Y. Feb. 14, 2014) ("Parties in litigation are required to be diligent and to act in good faith in producing documents in discovery ...[but] no one could or should expect perfection from this process."); see also Patrick Oot, 26(g): The Standard of Care When Responding to Discovery, The Federal Judges’ Guide to Discovery, p. 28 (2015). If the Federal Rules of Civil Procedure were previously only translucent on this point, it should now be clear with the renewed emphasis on proportionality.
As noted previously, at the outset, the parties have agreed to use key word searching. Dkt. # 153 (Case No. 17 CV 50107), Dkt. # 132 (Case No. 18 CV 379). The process of key word searching has pros and cons. The Sedona Conference Best Practices Commentary on the Use of Search & Information Retrieval Methods in E-Discovery, 15 Sedona Conf. J. 217, 231-34 (2014). And it is critical that key word searches are carefully created. Winfield, 2017 WL 5664852, at *7-8, 2017 U.S. Dist. LEXIS 194413, at *22-23 (citing William A. Gross Constr. Assocs. v. Am. Mfrs. Mut. Ins. Co., 256 F.R.D. 134, 136 (S.D.N.Y. 2009) ). With the advent of TAR and improvements made to that process, there appears to be a growing chorus that key word searching is not best practices, and that TAR is the way to go. Many trusted ESI experts - and courts - understandably sing the praises of TAR, especially its ability to scientifically quantify the successfulness of gathering and producing relevant documents. See Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182, 190-92 (S.D.N.Y. 2012); Maura R. Grossman & Gordon V. Cormack, Quantifying Success: Using Data Science to Measure the Accuracy of Technology-Assisted Review in Electronic Discovery, Data-Driven Law (2019). But that view is not universally held for all cases at this time. Raymond Biederman and Sean Burke, Biederman and Burke: Is use of keywords in e-discovery a game of "Go Fish"?, The Indiana Lawyer (November 16, 2016), available at https://www.theindianalawyer.com/articles/42021-use-of-keywords-in-e-discovery-a-game-of-go-fish ("In fact, the selective use of keyword searches can still be the cornerstone of an efficient and defensible search methodology. A properly deployed search takes advantage of a unique feature of electronic information - the ability to quickly and accurately find relevant people, words and concepts in large data sets."). And even as TAR becomes more acceptable and understood, in certain circumstances, key word searching and even linear review are not necessarily unreasonable. Timothy T. Lau & Emery G. Lee, Technology-Assisted Review for Discovery, p.7 (Federal Judicial Center 2017) ("For all its sophistication, TAR is not always better than search terms at identifying responsive documents."); Shannon Capone Kirk & David J. Kessler, The Use of Advanced Technologies in Document Review, The Federal Judges’ Guide to Discovery, p. 80 ("It is not unreasonable to use search terms or to review documents manually.") (emphasis in original). Indeed, when being pitched on the virtues of TAR, some parties, attorneys and courts may feel - albeit without any evidence-based reason - as though they are being sold a monorail. And, most importantly, the parties in this case have agreed to use key word searching at the outset. Dkt. # 153 (Case No. 17 CV 50107), Dkt. # 132 (Case No. 18 CV 379). This Court will not micromanage the litigation and force TAR onto the parties. See Craig Shaffer, Deconstructing "Discovery About Discovery", 19 Sedona Conf. J. 215, 225 (2018) ("A judge must guard against unilaterally usurping the discovery process or delving into technical areas beyond their control or expertise."). Courts cannot preach party cooperation and then unilaterally reject the reasonable process the parties agreed to use. So, for now, the parties will use key word searching.
The Simpsons, Marge vs. the Monorial, https://en.wikipedia.org/wiki/Marge_vs._the_Monorail (last visited Aug. 7, 2018).
With key word searching (as with any retrieval process), without doubt, relevant documents will be produced, and without doubt, some relevant documents will be missed and not produced. That is a known known. The known unknown is the number of the documents that will be missed and not produced.
But there is a process by which to determine that answer, thereby making the known unknown a known known. That process is to randomly sample the null set. Karl Schieneman & Thomas C. Gricks III, The Implications of Rule 26(g) on the Use of Technology-Assisted Review, 2013 Fed. Cts. L.Rev. 239, 273 (2013) ("[S]ampling the null set will establish the number of relevant documents that are not being produced."). Consequently, the question becomes whether sampling the null set is a reasonable inquiry under Rule 26(g) and proportional to the needs of this case under Rule 26(b)(1).
Rule 26(b)(2)(B), which specifically refers to ESI, does not appear applicable at this time. That rule provides that "[a] party need not provide discovery of electronically stored information from sources that the party identifies as not reasonably accessible because of undue burden or cost." Fed.R.Civ.P. 26(b)(2)(B) (emphasis added). The sources of the ESI do not appear to be the cause of any undue burden or cost at this time. Instead, Defendants contend that searching the null set would be costly and burdensome.
Sampling the Null Set is Reasonable Under Rule 26(g)
Conducting a random sample of the null set is a part of the TAR process. See DaSilva Moore v. Publicis Groupe, 287 F.R.D. 182, 202 (S.D.N.Y. 2012) (discussing "quality control by random sample of irrelevant documents"); see, e.g., Winfield v. New York, 15 CV 5236, 2017 WL 5664852, at *9, *11-12, 2017 U.S. Dist. LEXIS 194413, at *29, *35-36 (after a TAR review, requiring and describing the random sampling of non-responsive documents to be provided to the requesting party). The purpose of randomly sampling the null set after a TAR review is to validate the process and provide reasonable assurance that the production is complete. Commentary on Defense of Process: Principles and Guidelines for Developing and Implementing a Sound E-Discovery Process, p. 27 (September 2016); Charles Yablon & Nick Landsman-Roos, Predictive Coding: Emerging Questions and Concerns, 64 S.C. L.Rev. 633, 641 (Spring 2013). Validation and quality assurance are fundamental principles to ESI production. The process provides the reasonable inquiry supporting the certification under Rule 26(g).
Defendants provide no reason establishing that a random sampling of the null set cannot be done when using key word searching. Indeed, sampling the null set when using key word searching provides for validation to defend the search and production process, and was commonly used before the movement towards TAR. See Biederman & Burke, Biederman and Burke: Is use of keywords in e-discovery a game of "Go Fish?", The Indiana Lawyer (November 16, 2016), available at https://www.theindianalawyer.com/articles/42021-use-of-keywords-in-e-discovery-a-game-of-go-fish; Christina Zachariason, Predictive Coding - Putting the Cart Before the Horse, Law360 (May 3, 2012) ("And why wouldn’t the defendants have used predictive coding in the first place? Most likely because of the other tried and true search and retrieval measures at their disposal. The defendants provided that less than 5 percent of the documents reviewed each defendant’s ‘null set’ sample (the documents not hit upon in their search efforts) were deemed potentially responsive upon additional QC. This would mean that 95 percent were not relevant and therefore appropriately cast aside. Certainly that should be considered a reasonable result under the rules. Use of established search and review methodologies is seemingly defensible assuming a reasonable result, that is, that the highest percentage of available relevant, nonprivileged documents be provided to the requesting party."); Craig Ball, Surefire Steps to Splendid Search, p. 7 (2009), http://www.craigball.com/Surefire_Steps_to_Splendid_Search.pdf ("Keyword search must be judged both by what it finds and what it misses. That’s the ‘quality assurance’ courts demand. A defensible search protocol includes limited examination of the items not generating hits to assess whether relevant documents are being passed over.") (emphasis in original). Just as it is used in TAR, a random sample of the null set provides validation and quality assurance of the document production when performing key word searches. Magistrate Judge Andrew Peck made this point nearly a decade ago. See William A. Gross Constr. Assocs., 256 F.R.D. at 135-6 (citing Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 262 (D. Md. 2008) ); In re Seroquel Products Liability Litig., 244 F.R.D. 650, 662 (M.D. Fla. 2007) (requiring quality assurance).
Accordingly, because a random sample of the null set will help validate the document production in this case, the process is reasonable under Rule 26(g).
Sampling the Null Set is Proportionate Under Rule 26(b)(1)
Defendants have only argued that a random sample of the null set would be expensive and burdensome. Dkt. # 154 (Case No. 17 CV 50107), Dkt. # 133 (Case No. 18 CV 379). Although Defendants do not specifically use the word "disproportionate," that is basically their argument.
There are two arguments Defendants are not making. First, Defendants do not argue that Plaintiffs will obtain other documents unrelated to this case that will somehow be used in future litigation and that the protective order in this case is insufficient to protect them. This is the "other bad stuff argument" (a phrase this Court has stolen from John Rabiej), which occurs when a party is concerned that a serial litigant or attorney will be able to develop other potential litigation based upon the ESI produced in the current litigation. Second, Defendants do not argue that documents produced after the sample of the null set will be irrelevant or nonresponsive.
The Court’s experience and understanding is that a random sample of the null set will not be unreasonably expensive or burdensome. Moreover and critically, Defendants have failed to provide any evidence to support their contention. Mckinney/Pearl Rest. Partners, L.P. v. Metro. Life Ins. Co., 322 F.R.D. 235, 242 (N.D. Tex. 2016) (party required to submit affidavits or offer evidence revealing the nature of the burden). Indeed, the Court’s experience and understanding is that the random sample will not be voluminous in the context of a case of this magnitude. See, e.g., EDRM Statistical Sampling Applied to Electronic Discovery, Sections 3 & 9, https://www.edrm.net/resources/project-guides/edrm-statistical-sampling-applied-to-electronic-discovery/ . If the Court is incorrect in this regard, Defendants can return to the Court and discuss its concerns more fully and supported by evidence.
Although Defendants have only argued expense and burden, when considering the other proportionality factors, the Court is persuaded that a random sample of the null set is appropriate in this case. Fed.R.Civ.P. 26(b)(1); see generally Revised Guidelines and Practices for Implementing the 2015 Discovery Amendments to Achieve Proportionality, 100 Judicature 21-33 (Winter 2016). First, the issues at stake in these cases are large. These cases allege a pharmaceutical company sells a critical drug prescribed for infants suffering from a rare seizure disorder at a cost of tens of thousands of dollars more than it previously sold for due to anticompetitive and racketeering behavior. Tellingly, for better or worse, these cases have garnered national media attention. 60 Minutes, The Problem with Prescription Drug Prices, https://www.cbsnews.com/video/the-problem-with-prescription-drug-prices/ (last visited Aug. 7, 2018). Second, the potential amount in controversy is extraordinary. The cases contain racketeering and antitrust claims, which, if successful, could lead to huge damage awards. In today’s legal vernacular, these are "bet the company" cases. Third, as noted previously, these cases involve asymmetrical discovery, with Defendants having access to the vast majority of the relevant information. Fourth, as to resources, the main defendant is a large international pharmaceutical company with substantial resources. Fifth, the ESI will play a key role in resolving the issues in these cases. Finally, the burden and expense of a random sampling of the null set does not outweigh its likely benefit of ensuring proper and reasonable - not perfect - document disclosure.
If, after the random sampling of the null set and producing any relevant documents identified by that process, Plaintiffs seek more documents and the parties cannot resolve the dispute, then the Court will again apply Rule 26(b)(1)’s proportionality principles to determine what, if anything, will be required of the producing party. And, at that point, the Court may be receptive to arguments about cost shifting if it is appropriate. See Fed.R.Civ.P. 26(c)(1)(B). Again, in determining what else needs to occur, the Court would want evidence relating to the proportionality factors, including possibly statistical evidence. See EDRM Statistical Sampling Applied to Electronic Discovery, Section 1, https://www.edrm.net/resources/project-guides/edrm-statistical-sampling-applied-to-electronic-discovery/ ("The stronger the case that further review would be expensive, fruitless and disproportionate, the better the argument for ending the review. Any decision to end review early needs to be backed up with appropriate facts that justify this choice and generally, no single factor will be determinative on its own. However, demonstrably valid statistics can be one of the factors used to justify this decision."); see also The Sedona Conference Commentary on Proportionality in Electronic Discovery, 18 Sedona Conf. J. 141 (2017).
Accordingly, the Court will require a random sample of the null set. Moreover, Plaintiffs’ proposed 95% confidence level with +/-margin of 2% is acceptable. Schieneeman & Gricks, The Implications of Rule 26(g) on the Use of Technology-Assisted Review, 2013 Fed. Cts. L.Rev. at 270 (citing Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182, 186 (S.D.N.Y. 2012) ).
D. CONCLUSION
The Court adopts the parties’ proposed order establishing the production protocol for ESI with the inclusion of Plaintiffs’ proposal that a random sample of the null set will occur after the production and that any responsive documents found as a result of that process will be produced. Moreover, following that production, the parties should discuss what additional actions, if any, should occur. If the parties cannot agree at that point, they can raise the issue with the Court.