Opinion
14-cv-680 (NRM) (PK)
09-16-2024
MICHAEL WALKER Plaintiff, v. THE CITY OF NEW YORK, GREGORY GORDON, and MICHAEL SMITH Defendants.
MEMORANDUM AND ORDER
NINA R. MORRISON, UNITED STATES DISTRICT JUDGE
This is a civil rights action brought under 42 U.S.C. § 1983, in which the Plaintiff, Michael Walker, alleges, inter alia, that New York Police Department (“NYPD”) officers Gregory Gordon and Michael Smith violated his constitutional right to be free from excessive force when they shot him in the back and seriously injured him during a police-citizen encounter in Staten Island in February 2013. Defendants do not deny that the officers shot Plaintiff but claim that Plaintiff was armed with a firearm and that their actions were justified to protect their own safety and that of the community. Plaintiff offers a starkly different account of their encounter: he maintains that he was unarmed, that the officers' decision to shoot him was unreasonable and unjustified, and that the officers proceeded to fabricate a false account of the incident to cover up their unlawful use of potentially deadly force.
In advance of a long-anticipated jury trial in this civil action, Plaintiff has moved to preclude Defendants the City of New York, Gregory Gordon, and Michael Smith from offering at trial evidence generated by the Office of the Chief Medical Examiner (“OCME”) using its Low Copy Number (“LCN”) testing and Forensic Statistical Tool (“FST”) on a mixed DNA sample consisting of 22.35 picograms (pg) of DNA. The DNA sample in question was swabbed by OCME from a “trigger and trigger guard” of a firearm that was reportedly recovered near the scene of Plaintiff's shooting.
In a docket order dated September 11, 2024, the Court granted Plaintiff's motion to preclude with “additional reasoning supporting its ruling” to follow. Order dated Sept. 11, 2024. The Court now writes to provide its reasoning.
As explained below, Defendants, as the proponents of this scientific evidence, have not met their burden of showing that the evidence is reliable and admissible under Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). Specifically, Defendants have failed to demonstrate the reliability of OCME's methods for estimating the number of contributors to a non-deconvoluted DNA mixture where, as here, the mixture is below 25 pg in quantity. Moreover, because the estimation of contributors is a threshold determination that substantially impacts the “likelihood ratio” generated by the proprietary software, FST, that OCME uses to explain the potential statistical significance of its conclusions to the jury, the OCME's analysis and conclusions as to the DNA sample in this case are inadmissible.
BACKGROUND
I. Forensic DNA Analysis and OCME Methodologies
The fundamentals of forensic DNA testing are not at issue in this case, nor are the parties in disagreement over them. For the context in which the parties' dispute over the specific DNA testing methods used here arises, they each cite a brief overview of the science of DNA testing by Dr. Craig O'Connor, Defendants' expert witness and a Deputy Director in OCME's Department of Forensic Biology:
Forensic DNA profiling targets areas of DNA that vary from person to person. DNA testing can determine these genetic markers and compare biological samples from different individuals. Short Tandem Repeats or “STR[s]” are the specific groups of loci (locus in the singular) that are used to type and compare DNA. These are short segments of DNA that are repeated a number of times in tandem, or in a row. The length of the short segments of DNA that repeat along the DNA strand varies and can be used to type individuals.
DNA analysis looks at the length variation - specifically, at the number of repeats of the short segments - at an STR locus. The number of repeats at a particular location constitutes the DNA type or “allele” present at that location. For example, a “10” allele means that there were ten repeats of the sequence at that locus. Since each person receives half of their DNA from one biological parent and half from the other biological parent, each individual has two DNA alleles at each locus, each of which is identified by its length, determined by the number of repeats. An individual's DNA profile is essentially a string of numbers representing the number of repeats present at each location.
O'Connor Summary of Testimony at 6, ECF No. 239-20.
Basic forensic DNA analysis contains several steps. O'Connor Deposition, ECF No. 239-24. First, the criminalist extracts the DNA by adding chemicals to the sample that “break open the cells to release the DNA.” Id. at 15. Second, the amount of DNA in the sample is estimated through a series of tests in a process called quantitation or quantification. Id. If there is not enough or any DNA, the DNA analysis cannot continue. Id. However, if there is a threshold amount of DNA, the criminalist will move onto the third step, polymerase chain reaction (“PCR”) amplification. Id. at 15-16; United States v. Morgan, 53 F.Supp.3d 732, 736 (S.D.N.Y. 2014), aff'd, 675 Fed.Appx. 53 (2d Cir. 2017). The process of PCR amplification, which makes “millions and millions of copies of the DNA at the certain locations” needed for testing, O'Connor Deposition at 16, can result in random error, or “stochastic effects,” id. at 22-24. Stochastic effects include: allelic dropout, where alleles fail to appear in the profile; allelic drop in, where alleles not originating from the principal donors show up in the DNA profile; and increased stutter, which is a biological artifact that takes place during the amplification process. Id. at 24-26. After the amplification process, the DNA goes through a process called capillary electrophoresis, during which DNA fragments are separated out by size. Id. at 16. OCME uses a genotyping or genetic analyzing program to visualize the results, and analysts can then interpret the results and make conclusions. Id. at 16-17.
In this case, the DNA sample taken from the firearm contained two distinct characteristics relevant to its analysis. First, it was below 100 pg, and second, no distinct (or “major”) DNA profile could be deduced from the sample. Accordingly, the OCME used two specific tools to test and analyze the DNA sample: LCN and FST.
i. Low Copy Number (“LCN”)
Dr. O'Connor explained that LCN is a term of art that refers to the analysis and interpretation of low-level samples of DNA. O'Connor Deposition at 36. LCN provides a “more sensitive way of analyzing lower amounts of DNA and applying modified interpretation procedures to account for” the increased stochastic effects that are expected with low samples of DNA. Id.
The court in Morgan provided the following overview of LCN testing:
OCME . . . uses its LCN DNA testing protocols when the amount of source material is less than 100 picograms (“pg”) of DNA, as distinguished from what it calls high copy number (“HCN”) DNA
analysis, which uses samples containing more than 100 pg of DNA. . . .OCME's LCN testing uses the same basic steps as HCN testing: (1) extraction of DNA from the sample (e.g., blood, bone, hair, saliva, semen, or skin cells), (2) quantitation of the amount of DNA extracted from the sample, (3) amplification of the DNA using polymerase chain reaction (“PCR”), and (4) analysis. The primary differences between HCN and LCN DNA testing are at the stages of amplification and analysis.
The difference in the amplification step is that HCN DNA testing employs 28 “rounds” of PCR amplification whereas LCN DNA testing employs 31 rounds. The small quantity of starting material in conjunction with the increased number of rounds of PCR can result in an increase in “stochastic effects,” which are random errors that create inaccuracies in DNA testing.Morgan, 53 F.Supp.3d at 735-36 (footnote and citation omitted).
In addition to three additional cycles of amplification, OCME amplifies the low-template sample in triplicate, meaning that it “take[s] the sample and put it in three different tubes for that amplification procedure.” O'Connor Deposition at 39.
OCME developed the LCN method in-house. It began performing validation studies - “the process through which a procedure is evaluated to determine its efficacy and reliability for forensic casework” - for its LCN DNA testing in 2003 and 2004. Morgan, 53 F.Supp.3d. at 737. In accordance with the guidelines created by the Scientific Working Group of DNA Analysis Methods (“SWGDAM”), OCME used known samples in performing validations. Id. at 737-38. In testing various singlesource DNA samples - that is, samples that contained (unlike in the instant case) DNA from a single contributor - OCME's analysts successfully determined 92 percent of all alleles for samples containing between 150 and 25 pg. Id. at 738. When the quantity of DNA in the sample decreased below 25 pg, however, the accuracy rate decreased; OCME's analysts successfully determined 77 percent and 51 percent of all expected alleles for single-source samples containing 12.5 and 6.25 pg, respectively. Id. OCME also “performed LCN testing on mixture samples containing two DNA contributors,” with the goal of determining “whether OCME's testing could accurately ascertain the DNA profile of the ‘major contributor' - the contributor with the larger presence of DNA in the sample.” Id. Based on the results of these validation studies, “OCME created its interpretation guidelines, intended to allow for consistent interpretation of LCN testing results by accounting for the presence of increased stochastic effects as the quantity of DNA decreases.” Id.
In 2005, the DNA Subcommittee of the New York State Commission on Forensic Science (“the Commission”) approved OCME's use of LCN in forensic casework. Id. at 739. Both prior to and following that approval, however, LCN's reliability has been the subject of considerable debate among forensic DNA experts and numerous court challenges, both as to LCN's general protocols and its application to the analysis of specific kinds of DNA samples that, these challengers contend, are especially difficult to interpret. For example, New York's highest court held in 2020 that - notwithstanding the Commission's approval in 2005 of LCN for use in forensic casework - a New York state trial court abused its discretion by declining to hold a Frye hearing to fully consider a criminal defendant's challenge to OCME's performance of LCN testing on a DNA mixture from at least two contributors. People v. Williams, 35 N.Y.3d 24, 30-31, 38 (2020). The Williams court noted, among other things, that the defendant had raised “sufficient questions regarding the general acceptance of LCN evidence, based on its lack of use by other laboratories, the absence of prior meaningful review, and the scientific article proffered by defendant regarding the reliability of the evidence for criminal prosecution” as to entitle him to a “have the People put to [their] burden” of affirmatively proving its admissibility. Id. at 40.
Nearly two decades after its development, OCME remains the only DNA laboratory in the United States that uses LCN in forensic casework. Krane Report at 18, ECF No. 234-12; United States v. Cortorreal, 668 F.Supp.3d 309, 316 (S.D.N.Y. 2023). While the University of North Texas Health Science Center performs LCN testing, it only does so “for missing persons identification.” Williams, 35 N.Y.3d at 33.
In an apparent attempt at rebranding, at some point OCME began to refer to LCN not as “low copy number” testing but instead as “high sensitivity testing.” O'Connor Deposition at 36. But as the great songwriter Dolly Parton noted in the different (and far more familiar) context of recovering from heartbreak, simply using an opposite adjective to describe a phenomenon does not change its underlying reality. See Dolly Parton, The Grass Is Blue on The Grass is Blue (Sugar Hill Records 1999) (“I just can't make it/one day without you/Unless I pretend that the opposite's true/Rivers flow backwards/Valleys are high/Mountains are level/Truth is a lie/I'm perfectly fine/And I don't miss you/The sky is green/And the grass is blue”). Consistent with other courts that have discussed this form of testing, the Court uses the term Low Copy Number, or LCN.
ii. Forensic Statistical Tool (“FST”)
FST is a proprietary software developed by OCME. OCME developed its FST software “to enable the calculation of likelihood ratios (LRs) for samples involving two- and three-person mixtures where parts or all of the contributors are non-deducible,” meaning that “no distinct DNA profile can be determined.” O'Connor Summary of Testimony at 10. “FST can be used when a DNA analyst compares a reference profile with the profile or profiles contained in what has previously been determined to be either a two- or three-person mixture from a forensic sample.” Id. at 12. However, “FST is not validated for use in casework where the number of contributors is estimated to be greater than three.” Id. at 12 n.1.
To calculate a likelihood ratio, the analyst inputs what he or she has concluded is the likely number of contributors to the mixture into the FST tool. O'Connor Deposition at 48-49. Accordingly, the analyst must first review the data from the underlying DNA test results and estimate the number of contributors in the DNA sample. Id. at 47-49, 50. While “the true number of contributors will always be unknown in [a] casework sample,” id. at 59, a team of OCME scientists undertook a controlled study “[t]o develop guidelines to estimate the number of contributors to two-, three- and four-person mixtures containing either high template DNA [] or low templated DNA [] amounts,” Perez et al., Estimating the number of contributors to two-, three-, and four-person mixtures containing DNA in high template and low template amounts (the “Perez et al. Study”) at 1, ECF No. 234-6. To do this, the authors created 728 “purposeful” mixtures of blood or buccal samples from combinations of two, three, and four persons in amounts ranging from 10 to 500 pg, id. - that is, they intentionally created mixtures of DNA at various quantities of DNA and with the actual number of contributors known - so that they could study the properties of the DNA data generated when these hundreds of mixed samples were tested. The OCME researchers then noted the number of alleles in each of those mixtures and identified characteristics that were common to many of the mixtures of three and four persons. Id. at 1, 5. Those findings are set out in Table 2 of the Perez et al. Study and below.
(Image Omitted)
Id. at 5.
The OCME incorporated portions of the research from the Perez et al. Study into its protocols for estimating the number of contributors to a DNA mixture, which its analysts follow before those conclusions are entered into FST. Specifically, its protocols for STR analysis dated January 12, 2012 state that low template DNA samples “are considered three-person mixtures as follows: i. Five alleles are present in at least two loci in the consensus profile. ii. Stutter and other explainable artifacts should be considered when counting the number of alleles at a locus. iii. Inconsistencies among the replicates may indicate the presence of a third contributor.” Jan. 12, 2012 FST Protocols at 7, ECF No. 234-7. The protocols go on to state that “[f]or some three-person mixtures additional criteria may be explored,” and then set forth two tables adapted from Table 2 from the Perez et al. Study and note that in the study, the characteristics in those tables “were only observed in controlled mixtures with more than two contributors.” Id.
In addition to determining the estimated number of contributors, the analyst must manually determine if the individual who is the source of the reference sample (usually referred to as the “suspect” or “defendant” in forensic casework) is a possible contributor to the mixture. O'Connor Summary of Testimony at 12. FST then calculates a “likelihood ratio,” or LR. An LR “considers the probability of two separate hypotheses being true. Hypothesis A is the probability that the mixture contains the suspect's DNA. Hypothesis B considers the probability that the suspect is not a contributor to the mixture.” Id.
The Court here pauses to note that because such testing is most often conducted in ongoing criminal investigations, and the individual whose DNA is submitted for reference is often a person suspected and/or convicted of a crime, OCME's experts and various court decisions discussing LR methodology typically refer to these opposing hypotheses as “the prosecution's hypothesis” (in which the person's DNA is present in the mixture) and “the defendant's” or “suspect's” hypothesis (in which the person is not a contributor to the DNA). In this civil case, the parties are the Plaintiff, Michael Walker (whose hypothesis is that he is not a donor to the mixture of DNA swabbed from this firearm) and the Defendants, two individual officers and their employer, the City of New York (whose hypothesis is that Plaintiff is a donor to this DNA mixture).
FST is only intended to determine the likelihood of a particular outcome, and it “refines its estimate by taking into account several important factors including the drop-in/drop-out rates, and the possibility that the alleles in the sample could have been provided from another individual in the population based on estimated allele and genotypic frequencies.” Id. OCME also offers an interpretation of the strength or weakness of its calculation: if the likelihood ratio is “less than 10, it is considered to provide ‘limited support' for the prosecution's hypothesis; results of 10-100 show ‘moderate support,' 100-1,000 show ‘strong support,' and more than 1,000 show ‘very strong support' for the prosecution's hypothesis.” United States v. Jones, 965 F.3d 149, 155 (2d Cir. 2020); Adams Report at 36, ECF No. 234-16.
In December 2010, the New York State Commission on Forensic Science approved the use of FST. O'Connor Summary of Testimony at 12. In 2017, OCME replaced FST with more advanced software developed since FST came online, and no longer uses FST to perform statistical calculations in forensic casework. O'Connor Summary of Testimony at 11; O'Connor Deposition at 206. However, Dr. O'Connor contends that “OCME continues to stand by the validity of its FST analysis.” O'Connor Summary of Testimony at 11.
II. DNA Testing in Instant Case
In this 42 U.S.C. § 1983 action, Plaintiff Michael Walker alleges that Defendants, NYPD Officers Gregory Gordon and Michael Smith, used excessive force against him when they shot and injured him on February 2, 2013. Compl., ECF No. 1. While Plaintiff contends that he was unarmed when he encountered the officers, Defendants dispute that claim and argue that they shot Plaintiff after observing him with a firearm, under circumstances that they claim justified their use of such force.
A crime scene unit recovered a firearm on the street where Plaintiff was shot, and OCME analyzed a DNA sample that was taken from its trigger and trigger guard. OCME subjected the sample to two analyses. First, in or about August 2013, an OCME Level II criminalist, using OCME protocols including LCN testing, determined that (1) the firearm sample was 22.35 pg in quantity; (2) it was best described as a two-person mixture; and (3) it was suitable for comparison to suspected contributors. Pl. Mem. in Supp. at 4-5, ECF No. 234-1. Plaintiff contends that the criminalist's conclusion as to the number of contributors was apparently in accordance with OCME protocols because two loci had three or more repeating alleles and no loci contained five or more repeating alleles. Id at 18-19.
Second, in or about February 2014, the OCME criminalist used FST on the sample to determine a likelihood ratio. The criminalist concluded that it was “approximately 14,200 times more probable if the sample originated from Michael Walker and one unknown, unrelated person than if it originated from two unknown, unrelated persons.” Feb. 28, 2014 OCME Report at 1, ECF No. 239-3. The criminalist thus determined that there was “very strong support” that Plaintiff and one unknown, unrelated person contributed to the mixture, rather than two unknown, unrelated persons. Id.
Defendants seek to admit this evidence in support of their theory that Plaintiff was armed during the February 2, 2013 incident. Defendants agree that OCME's testing on this low-template, mixed DNA sample did not conclusively identify Plaintiff as a donor to the DNA swabbed from the firearm. But they maintain that OCME's testing and statistical analysis provides reliable evidence that they should be able to offer in support of their position on this disputed factual issue.
On August 14, 2024, Plaintiff filed a motion to preclude the introduction of evidence generated by the OCME on this sample under Federal Rules of Evidence 403 and 702. Pl. Mot. to Preclude, ECF No. 234. Defendants filed a response in opposition on August 23, 2024. Opp'n Br., ECF No. 239. While neither party requested to have their experts testify at an in-person Daubert hearing, they each submitted detailed briefing and an extensive array of exhibits regarding the testing methodologies used and the conclusions reached by OCME in this case for the Court's review, including expert witness reports, scientific articles, published and unpublished records of similar court proceedings, and deposition testimony. On September 4, 2024, the Court held a four-hour oral argument on the parties' pre-trial motions, the majority of which was devoted to argument on Plaintiff's motion to preclude the DNA evidence.
LEGAL STANDARD
Federal Rule of Evidence 702 allows admission of an expert witness's testimony
if the proponent demonstrates to the court that it is more likely than not that: (a) the expert's scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue; (b) the testimony is based on sufficient facts or data; (c) the testimony is the product of reliable principles and methods; and (d) the expert's opinion reflects a reliable application of the principles and methods to the facts of the case.Fed. R. Evid. 702. Importantly, “[w]hile the proponent of expert testimony has the burden of establishing by a preponderance of the evidence that the admissibility requirements of Rule 702 are satisfied, the district court is the ultimate ‘gatekeeper,'” and must determine that the expert's testimony “both rests on a reliable foundation and is relevant to the task at hand.” United States v. Williams, 506 F.3d 151, 160 (2d Cir. 2007) (citations omitted).
In Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993), the Supreme Court “enumerated a list of factors that, while not constituting a ‘definitive checklist or a test,' a district court might consider in evaluating whether a proffered expert opinion has the required indicia of scientific reliability” under Rule 702. Nimely v. City of New York, 414 F.3d 381, 396 (2d Cir. 2005) (quoting Daubert, 509 U.S. at 593-94). Those factors include whether (1) a theory or technique can be and has been tested; (2) it has been subjected to peer review and publication; (3) there is a high known or potential rate of error and there are standards controlling the technique's operation; and (4) the theory or technique enjoys general acceptance within a relevant scientific community. Kumho Tire Co. v. Carmichael, 526 U.S. 137, 149-50 (1999).
However, the Daubert factors “neither necessarily nor exclusively appl[y] to all experts or in every case,” and “the district court's inquiry into the reliability of expert testimony under Rule 702 is a ‘flexible one.'” Williams, 506 F.3d at 160 (citations omitted). The law thus “grants a district court the same broad latitude when it decides how to determine reliability as it enjoys in respect to its ultimate reliability determination.” Id. (emphasis in original). “In addition to the requirements of Rule 702, expert testimony is subject to Rule 403.” Nimely, 414 F.3d at 397. Accordingly, such evidence may be excluded “if its probative value is substantially outweighed by a danger of one or more of the following: unfair prejudice, confusing the issues, misleading the jury, undue delay, wasting time, or needlessly presenting cumulative evidence.” Fed.R.Evid. 403.
A court's obligation to ensure that the proponent of expert testimony has met its burden does not evaporate simply because that testimony concerns the well-regarded science of DNA analysis. As the New York Court of Appeals recently emphasized,
In the criminal justice system, [genetic biology] has provided forensic science with one of the most powerful tools for identification yet seen. DNA testing has become the ‘gold standard' of this process. For this reason, more than any other, courts must use the tools available to make sure the highest standards of reliability are maintained.Williams, 35 N.Y.3d at 29.
The prejudice that may result from the admission of insufficiently reliable methods of DNA testing and analysis is also well understood. That is particularly so as DNA laboratories attempt to test and analyze ever-more-minute quantities of evidence, ones that may include complex, difficult-to-interpret mixtures of DNA from multiple contributors. Further, as the President's Council of Advisors on Science and Technology (“PCAST”) noted in 2016, in an exhaustive and at times highly critical review of the scientific foundations of a range of forensic disciplines as currently employed in the criminal justice system, “testimony based on forensic feature-comparison methods poses unique dangers of misleading jurors” as “[t]he vast majority of jurors have no independent ability to interpret the probative value of results,” and “[t]he potential prejudicial impact is unusually high, because jurors are likely to overestimate the probative value of a ‘match' between samples.” Executive Office of the President PCAST, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods, 45 (Sept. 2016), https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcas t_forensic_science_report_final.pdf (the “PCAST Report”).
The admission of insufficiently validated or otherwise unreliable applications of DNA testing and analysis poses additional risks. That is because, despite a common assumption that DNA evidence is infallible, inherent subjectivity pervades “even the most favorable conditions of forensic DNA typing.” Erin Murphy, The Art in the Science of DNA: A Layperson's Guide to the Subjectivity Inherent in Forensic DNA Typing, 58 Emory L. J. 489, 509 (2008). Those concerns have particular resonance where, as here, the proffered evidence is a mixture of DNA from multiple individuals, and the sum total of DNA present in that mixed sample is extremely low. Indeed,
Readers who lack formal scientific training or who are otherwise less than familiar with some of the specific interpretive challenges that arise in the context of modern DNA analysis may wish to review Professor Murphy's excellent essay in full. It includes, inter alia, cogent explanations of the challenges that arise in DNA interpretation through analogies to other contexts that are more familiar to nonscientists, and also explains why “the use of DNA typing to inculpate a person . . . fundamentally differs from its use to exculpate.” Murphy, supra at 493. For a more detailed treatment of these and related topics, see also Erin E. Murphy, Inside the Cell; The Dark Side of Forensic DNA (2015).
The fundamental difference between DNA analysis of complex mixture samples and DNA analysis of single-source and simple mixtures lies not in the laboratory processing, but in the interpretation of the resulting DNA profile.
DNA analysis of complex mixtures is inherently difficult. Such samples result in a DNA profile that superimposes multiple individual DNA profiles. Interpreting a mixed profile is different from and more challenging than interpreting a simple profile, for many reasons. It is often impossible to tell with certainty which genetic variants are present in the mixture or how many separate individuals contributed to the mixture, let alone accurately to infer the DNA profile of each one.
PCAST Report. supra at 7-8; see also id. at 75-76 (explaining numerous factors that make it difficult for analysts to reliably interpret mixed-donor DNA samples, and why those factors are “even more [present] for small amounts of DNA”).
Accordingly, as with any other scientific evidence whose admission is challenged by an opposing party, the proponent's use of DNA evidence and the specific conclusions offered by its DNA experts must be screened for scientific validity and reliability. This requires courts to determine, among other things, whether foundational principles of scientific validity support the DNA expert testimony being offered. As with other disciplines, to constitute valid and reliable science, the methods of DNA testing or analysis at issue must be “subjected to empirical testing, under conditions appropriate to its intended use, that provides valid estimates of how often the method reaches an incorrect conclusion.” Id. at 46.
DISCUSSION
Plaintiff makes five arguments in support of his motion to preclude the DNA evidence in this case: (1) the methodologies used by OCME have not been shown to be a reliable means of accurately determining the number of contributors to low-level mixtures of DNA such as the firearm sample at issue here; (2) FST has not been sufficiently validated for mixtures of DNA whose total quantitation is 25 pg or below; (3) OCME's conclusion that there is “very strong support” for the hypothesis that Plaintiff and an unknown person contributed to the firearm is misleading and unfairly prejudicial; (4) the DNA testing should be excluded in light of OCME's failure to compute a likelihood ratio as to a second individual (the “confidential source”), from whom a DNA reference sample was also submitted for testing; and (5) FST's design and development were flawed, resulting in a software program that does not comport with accepted methodologies of software engineering.
For the reasons to follow, the Court agrees with Plaintiff's first argument - namely, that Defendants have failed to demonstrate the reliability of OCME's methods for estimating the number of contributors to a mixed sample whose total quantity of DNA is below 25 pg. The DNA evidence and expert testimony that Defendants seek to offer in this case - OCME's interpretation of the DNA data in that sample, and its opinions as to the statistical likelihood that Plaintiff is (or is not) one of the contributors - are thus unreliable. Accordingly, the Court need not reach Plaintiff's remaining challenges to the admissibility of this evidence.
I. Reliability of OCME's Protocols for Estimating Contributors to a Mixture
Plaintiff argues that the FST “requires the OCME examiner to input the number of contributors to the DNA it is analyzing: one, two or three,” but no entity “has ever validated the protocols the OCME uses to estimate the number of contributors for samples that, as here, are quantitated at 25 [pg] and below.” Pl. Mem. in Supp. at 12. The Court agrees.
As an initial matter, other than the Perez et al. Study, which appeared to develop and test aspects of the protocols that OCME relies on for estimating contributors, Defendants have failed to point to any validation studies that have tested the accuracy or reliability of their methods for estimating the number of contributors to an extremely low-level (under 25 pg) DNA sample like this one. When asked whether he was “aware of other research that has been conducted to identify if the characteristics used by OCME to differentiate two-person from a three-person, or even a four-person sample, in fact reliably do so,” Dr. O'Connor replied that he could not think of any “off the top of [his] head.” O'Connor Deposition at 126-27. At oral argument, the Court reiterated this question to defense counsel, who confirmed that they are not aware of any other studies that have tested OCME's ability to accurately identify the number of contributors using its current protocols for samples below 50 pg. Nor did Defendants identify any such studies or data in their post-argument submissions. See Letter dated Sept. 7 at 1, 2024, ECF No. 251.
Instead, Defendants ask the Court to rely on the general validation studies for LCN and FST. They contend that “OCME's protocols for estimating the number of contributors to a DNA mixture were documented in the validation materials turned over to the NYS Commission on Forensic Science DNA Subcommittee, as part of the approval process for OCME's LCN testing protocols.” Id. But as the New York Court of Appeals recognized in Williams, 35 N.Y.3d at 41, the DNA Subcommittee's approval - while relevant to the general acceptance inquiry - “is no substitute for the scrutiny of the relevant scientific community.” The Williams court reasoned that sole reliance “on the Subcommittee's approval as dispositive of the general acceptance would [] supplant the courts' obligation to ensure, under Frye, that scientific techniques and methods are sufficiently reliable to be admitted into evidence.” Id. Accordingly, the court determined that the trial court abused its discretion by admitting DNA evidence where LCN and FST were used without a Frye hearing. Id. at 29-30.
While the Williams court reached this conclusion in the context of a Frye, rather than Daubert analysis, the Court finds persuasive the Williams court's conclusion that the DNA Subcommittee's approval cannot, on its own, demonstrate the reliability of a scientific technique. That is particularly so where, as here, Defendants point to no evidence that, in considering whether to approve OCME's use of LCN and FST more generally, the DNA Subcommittee ever considered whether those methods of estimating the number of contributors could be validly applied to a mixed sample of 25 pg or below, including where, as here, the mixture could not be deconvoluted.
On the other hand, Plaintiff argues that the findings of the Perez et al. Study bear heavily on the specific question posed in this case. That study indeed appears to be the only controlled study testing OCME's ability to reliably estimate the number of contributors to a mixed DNA sample using the methodologies employed here (LCN and FST). As discussed supra at 8-9, the authors of that study created 728 purposeful mixtures of two-, three- and four-person mixtures and then noted the number of alleles in the mixtures and compiled certain characteristics - as reflected in Table 2 of the study - that were common among three- and four-person samples. Perez et al. Study at 5. The authors then evaluated whether the observed mixture characteristics were present in 117 mixtures generated from items handled by two, three, and four individuals. Id. at 5, 11.
The Perez et al. Study makes clear that the criteria the authors identified as indicative of two-, three-, and four-person mixtures - many of which were which were subsequently incorporated into OCME's protocols - were substantially less reliable predictors when applied to low template DNA samples. For example, while “template amounts less than 25 pg were not amplified for three-person mixtures,” id. at 10, the study states that several 25 pg three-person mixtures did not meet the Table 2 criteria for a three-person mixture, Id. at 8. Additionally, only 25 of the 50 low template DNA “four-person samples looked like four-person samples by the total number of different alleles labeled and/or by the patterns listed in Table 2,” and “the vast majority of the samples that did not meet the four-person criteria contained less than 50 pg of template DNA.” Id. at 10.
Moreover, while the results of the study suggest “that samples with 49 or fewer alleles are best described as two-person mixtures, with 52 to 59 alleles as three-person mixtures, and with 65 or more alleles as four-person mixtures,” Id. at 5, “[m]ixtures with template amounts of 10 pg and 15 pg made with DNA from 4 contributors contained 33-51 different alleles suggesting that there was so much drop out that they were actually composed of fewer than 4 persons,” Id. at 10. Additionally, of the sixteen four-person samples that were amplified with 20 to 40 pg of template DNA, none “showed more than 66 different alleles and only 2 would be called four-person mixtures by the criteria in Table 2.” Id. In other words, at least some of the samples under 25 pg that were known by the study's authors to contain DNA from four persons yielded so little DNA data that they appeared to have fewer contributors than they actually did. The study also notes that the number of alleles in four-person samples “var[ies] dramatically depending upon the template amount for samples with less than 50 pg.” Id. at 6.
By contrast, the Perez et al. Study indicates that the criteria the authors identified for estimating contributors are more reliable with higher template DNA amounts. For example, the study notes that “[f]our-person mixtures with at least 50 pg of template DNA presented fewer challenges.” Id. at 10. Unlike the samples with 20 to 40 pg of template DNA, where at least one contributor to 15 of the 16 samples was missing more than two alleles, none of the contributors to 24 of the 30 four-person mixtures with at least 50 pg of template DNA was missing two or more alleles. Id.
The study's authors summarized that, overall, “[u]sing the total number of different alleles and the characteristics listed in Table 2, 86% of all [high template] DNA and 75%-77% of all 50-100 pg [low template] DNA four-person purposeful mixtures resembled four-person mixtures.” Id. at 12. Importantly, however, this “was not the case for almost all of the samples with less than 50 pg of template DNA.” Id. Rather, for this under-50-pg category of four-person samples (a/k/a the “very low template” samples), the reliability of the characteristics used by OCME to determine the number of contributors fell dramatically. For “[w]hen the profiles of the true contributors were examined, the very low template samples showed extreme dropout and could better be described as three-person or even two-person mixtures.” Id.
Despite these varied results for samples below 100 pg, OCME continues to apply the same protocols for estimating contributors for all low template (i.e., below 100 pg) samples, regardless whether those samples are 50 pg, 25 pg, or below. See Jan. 12, 2012 FST Protocols at 6-8.
Taken together, the results from the Perez et al. Study and the otherwise limited validation of this aspect of OCME's DNA testing and interpretation protocols demonstrate that OCME's methods for estimating the number of contributors for DNA samples below 25 pg lack the required indicia of scientific reliability under Daubert. That is, the technique has been subject to limited testing, peer review, and publication. And the testing that has been conducted indicates that the technique carries a high rate of error when, as here, it is applied to DNA samples below 25 pg.
As indicated, there is substantial evidence in the record indicating that OCME's methods are unreliable for estimating contributors to samples 50 pg in quantity or less. However, given that the DNA sample here was below 25 pg, the Court need not decide the outer parameters of this method's reliability.
II. Impact of OCME's Determination of Number of Contributors on its Likelihood Ratio Calculation
In addition to showing that OCME's methods for estimating contributors to a DNA mixture below 25 pg is unreliable as a general matter, Plaintiff casts doubt on the OCME's conclusion that the DNA sample swabbed from the firearm in this case was a two-, rather than three- (or more) person mixture.
The OCME laboratory report dated August 29, 2013 states that “[a] mixture of DNA from at least two people was found” for the firearm sample. Krane Report at 2. However, Plaintiff's expert, Dr. Dan Krane, independently reviewed OCME's testing data and reached a different conclusion. According to Dr. Krane, the fact that across all three sets of genotyping results for the firearm sample, five different alleles are observed at two loci, indicates that there were at least three contributors to the sample. Id. at 7. And while four alleles were observed at the FGA locus, Plaintiff has a 26 allele at that locus, which was not observed in any results at the FGA locus. Id. Accordingly, if Plaintiff had been a contributor, his allele at the FGA locus would have had to drop out in each of the three separate rounds of DNA testing conducted by OCME on this sample. And according to Dr. Krane, if Plaintiff's DNA had been present in the sample, the FGA locus would have contained at least five alleles (i.e., indicating the presence of DNA from more than two donors). Id. It is thus Dr. Krane's opinion that “the simplest explanation of the genotyping results” of the firearm sample “is that it is a partial profile of DNA from three (or more) contributors.” Id. at 10.
Defendants attempt to evade preclusion of the DNA evidence by arguing that, even if Plaintiff is correct and the sample actually contains three (or more) contributors, any underestimation of the number of contributors typically results in the lowest possible likelihood ratio. This scenario, they argue, would benefit Plaintiff, because - even though OCME never calculated the actual LR for this scenario - the likelihood that Plaintiff and two unknown persons are contributors to the DNA sample would likely be even higher than the 1 in 14,200 LR that OCME generated using FST with its two-donor assumption. Opp'n Br. at 28.
Defendants also attempt to undermine Plaintiff's critique of this methodology by arguing that the number of true contributors to a forensic sample can never be known. This is a red herring. “Admissibility under Rule 702 does not require perfect methodology.” Hollman v. Taser Int'l Inc., 928 F.Supp.2d 657, 669 (E.D.N.Y. 2013) (quoting Best v. Lowe's Home Ctrs., Inc., 563 F.3d 171, 181 (6th Cir. 2009)). Rather, the expert testimony “need only rest on a reliable foundation that is relevant to the task at hand.” Hart v. Rick's Cabaret Int'l, Inc., 60 F.Supp.3d 447, 467 (S.D.N.Y. 2014). Defendants have not met their burden of showing reliability of the methodology as applied to this particular DNA sample because they have failed to show that such a reliable foundation exists when it comes to OCME's methods for estimating contributors (and the resulting likelihood ratio) to a mixed DNA sample of 25 pg or below.
However, the parties' experts agree “that the effect on the [likelihood ratio] within FST is not specifically known when the number of contributors assigned to a sample is an underestimate of the true value.” O'Connor Response at 6, ECF No. 234-2 (emphasis supplied). Defendants cite studies indicating that an underestimate of contributors may lead to a lower likelihood ratio - but Plaintiff cites cases and studies supporting the opposite conclusion. Moreover, certain evidence cited by Plaintiff demonstrates the substantial prejudice that can result when OCME underestimates the number of contributors to a mixed DNA sample.
For example, in United States v. Cortorreal, 17-cr-438 (S.D.N.Y.), the government sought to introduce evidence of DNA analysis using FST on a sample taken from a swab from a piece of duct tape which the OCME initially concluded was best described as a three-person mixture. See Gov't Opp'n to Mot. to Exclude at 4, United States v. Cortorreal, 17-cr-428 (S.D.N.Y. Sept. 7, 2021), ECF No. 644. Under that assumption, FST generated a likelihood ratio in excess of 10,000. Id. The remainder of the swab was later re-run using newer technology, STRMix - at which time the analyst concluded that the sample should instead be characterized as a mixture of four persons. Letter to the Court, United States v. Cortorreal, 17-cr-438 (S.D.N.Y. Feb. 23, 2023), ECF No. 773. Under the assumption that the sample was a four-person mixture, the OCME determined that “the likelihood ratio was in the uninformative range.” Id. The fact that increasing OCME's estimate of the number of contributors from three to four in a mixed sample in Cortorreal resulted in such a dramatic decrease in the likelihood ratio - from one in more than 10,000, all the way down to “uninformative” - is powerful evidence that underestimating the number of contributors to a low-template DNA sample may not just be incorrect; it may be highly prejudicial to a suspect when the likelihood ratio is calculated. Id. Plaintiff also cites an article that describes a case study involving two suspects who both denied committing the offense in question, and in which a likelihood ratio for one of the suspects was “much larger if two contributors are analyzed” than three contributors. Gill & Haned, A new methodological framework to interpret complex DNA profiles using likelihood ratios (“Gill Article”) at 6, ECF No. 234-13.
After the aforementioned retesting, the government decided not to offer the DNA evidence from the duct tape swab. In a subsequent opinion, the court in Cortorreal, 668 F.Supp.3d at 316, held that LCN DNA test results on a different piece of evidence were admissible where (unlike in the instant case) OCME was able to deconvolute a major donor profile. Id. at 316, 319. The court noted that it “remain[ed] troubled by certain aspects of OCME's methodology for determining the number of contributors to non-deconvolutable mixtures,” but declined to reach the admissibility of such evidence because the government withdrew the DNA evidence from the non-deconvolutable mixture. Id. at 321 n.18.
Given the very real possibility that an erroneous estimation of contributors could taint an FST-generated likelihood ratio, and that the impact of such errors would not be “conservative,” Defendants have not met their burden of demonstrating the reliability of OCME's conclusions in this case.
Plaintiff also emphasizes the possibility that there were four contributors to the mixture. Were that the case, the use of FST - which has not been validated for use where the number of contributors is estimated to be greater than three - would have been unsupported. While Dr. Krane does not highlight specific characteristics of the firearm sample that are consistent with a four-person sample, the Perez et al. Study makes clear that many of the four-person samples under 50 pg “showed extreme drop-out and could better be described as three-person or even two-person mixtures.” Perez et al. Study at 12. Plaintiff also points out that the overall number of alleles detected in the firearm sample is 45, which is similar to the 48 alleles detected in another extremely low DNA template (15 pg) four-person sample in the Perez et al. Study. Id. at 6, Figure 3. In light of those results, and the well-recognized principle that stochastic effects increase with lower amounts of DNA, see O'Connor Deposition at 25-26, the possibility of a fourth contributor to this sample cannot be dismissed as merely remote or speculative. That possibility further favors exclusion of the DNA evidence in this case.
III. Other Courts' Treatment of LCN and FST
Finally, Defendants contend that the DNA evidence should be admitted because “plaintiff points to no case where a Court has sustained a Daubert challenge to evidence of an FST analysis of low-template DNA.” Opp'n Br. at 22. But while that may be true, Defendants point to no case where a Court has denied a Daubert challenge in this context. In other words, Defendants have identified no case in which any court has found that that an FST-generated likelihood ratio produced from a mixed sample of 25 pg of DNA or less is sufficiently reliable under Daubert to be admitted into evidence at trial. Moreover, the cases on which Defendants rely to contend that “the vast majority of courts to have considered challenges to” LCN and FST “have admitted the evidence” are materially distinguishable. Id. at 23.
To start, Defendants cite United States v. Jones, No. 15-cr-153, 2018 WL 2684101 (S.D.N.Y. June 5, 2018), aff'd, 965 F.3d 149 (2d Cir. 2020). In that case, the defendant moved to exclude evidence of OCME's conclusion, through its use of FST, that the data obtained from a DNA mixture “is approximately 1340 times more probable if the sample originated from [defendant] and two unknown, unrelated persons than if it originated from three unknown, unrelated persons.” Id. at *6 (emphasis omitted). The court concluded that “expert testimony on the FST in this case rests on a reliable foundation and is relevant to the task at hand” and denied the defendant's motion. Id. at *12. However, that holding has no bearing on the instant analysis because (1) the court did not address OCME's protocols for estimating the number of contributors, and (2) there is no indication that the testing in Jones involved a low-template DNA sample, nor that OCME used LCN to test and analyze the data from that sample.
Next, Defendants point to Morgan, 53 F.Supp.3d at 736, 747, wherein the court rejected a defendant's challenge to OCME's use of LCN testing on a 14 pg sample of DNA. The defendant lodged numerous objections to the reliability of the evidence, including that the analyst incorrectly concluded that the sample contained DNA from two or more contributors. Id. at 746. In defendant's view, the sample was more accurately described as containing DNA from three or more people, and because OCME's mixture validation studies tested samples with only two contributors, the analysis of this sample was unreliable. Id. at 745. The court disagreed with that argument because OCME's protocols “give analysts discretion in determining the number of contributors,” and the determination of the number of contributors in that case “was clearly within the analyst's permissible discretion and consistent with OCME's protocols.” Id. at 745-46.
However, in Morgan, unlike here, OCME was able to discern a major contributor from the mixture; thus, it did not rely on FST to reach a conclusion regarding the key issue of what statistical weight (if any) the jury should give to the prosecution's hypothesis that Morgan may have contributed to the DNA sample.OCME's determination regarding the number of contributors in Morgan was not nearly as critical to its ultimate conclusions as it is when OCME utilizes FST. With FST, the determination of contributors is a critical input for calculating the likelihood ratio - which is “only as reliable as the predicate assumptions integrated into the FST software program.” Williams, 35 N.Y.3d at 52 (DiFiore, C.J., concurring). Moreover, while the Second Circuit held that the Morgan court did not abuse its discretion in admitting this evidence, the Court took pains to emphasize that “LCN analysis is supported by significantly weaker evidence of reliability than traditional DNA analysis,” “express[ed] no opinion on the propriety of admitting the results of LCN testing in other cases,” and “note[d] that OCME is discontinuing its use of LCN testing in favor of newer technology.” United States v. Morgan, 675 Fed.Appx. 53, 5556 (2d Cir. 2017). The Second Circuit's opinion in Morgan is thus far from a ringing endorsement of LCN testing and, in any event, does not address the reliability of OCME's methods where both LCN and FST are utilized.
The district court's opinion does not mention whether OCME calculated a likelihood ratio, nor any other statistical measure through which the jury could assess the degree of scientific support for the prosecution's hypothesis that Morgan was a donor to the DNA in question. And the Second Circuit's order noted only that OCME had concluded that LCN testing “showed the presence of DNA consistent with Morgan's genetic profile” on the evidence tested. Morgan, 675 Fed.Appx. at 54.
Defendants also rely on the recent decision by a trial court in Kings County to admit DNA evidence that used both LCN testing and FST in People v. Burrus, 200 N.Y.S.3d 655, 731 (N.Y. Sup. Ct. 2023). As an initial matter, the Burrus court considered whether the evidence was admissible under the Frye standard, which is solely concerned with whether the methodology is generally accepted in the relevant scientific community. Id. at 714; Frye v. United States, 293 F. 1013 (D.C. Cir. 1923). By contrast, under Daubert, courts can consider a variety of factors, including but not limited to whether the technique enjoys general acceptance within a relevant scientific community, to aid its determination of whether the evidence is reliable. While some courts have described the Daubert standard as more “liberal,” Nimely, 414 F.3d at 395-96, “[c]ommentators have extensively debated which test is the stricter standard,” Edward K. Cheng & Albert H. Yoon, Does Frye or Daubert Matter? A Study of Scientific Admissibility Standards, 91 Va.L.Rev. 471 (2005). Regardless of how that debate unfolds, a court's gatekeeping obligations under Daubert indisputably differ in kind from its obligations under Frye. Accordingly, the Burrus court's decision to admit evidence under Frye is only persuasive with respect to one of several factors this Court considers under Daubert.
But even in the context of the general acceptance factor, the Court finds the Burrus decision to have limited persuasive value because it did not consider the precise question at issue here: whether OCME's methods for estimating contributors to a DNA sample of 25 pg or less are generally accepted as reliable. Rather, the Burrus court's decision to admit the evidence appears to rest on its conclusion that LCN and FST are “not some bogus testing, and certainly not junk science.” 200 N.Y.S. at 723. Respectfully, this Court does not share the view that an analysis of the general acceptance of a methodology starts or ends with an inquiry into whether that methodology amounts to “junk science.” It is certainly true that courts are obligated - under both Daubert and Frye - to exclude expert evidence whose foundation is so patently flimsy as to not even be worthy of the term “science.” But that is not the outer limit of a court's gatekeeping function. Instead, proponents of scientific evidence may be able to show that a particular method is generally accepted by the relevant scientific community or otherwise reliably applied in one context yet fail to meet their burden when that methodology is used in another. This case does not require the Court to consider (much less decide) whether OCME's use of LCN and/or FST on low-template DNA mixtures can ever meet that threshold. For the present purposes, the Court notes only that there is a vast gulf indeed between the evidence OCME has offered demonstrating the acceptance and reliability of its methods for testing robust DNA samples from either a single source or mixtures with a distinctly-identifiable major donor on the one hand, and its methods for interpreting extremely low quantities of DNA in mixed DNA samples on the other.
Importantly, Defendants could not point to any other cases involving Frye or Daubert challenges to DNA evidence where the “evidence consisted of a non-deducible DNA mixture, under 25pg, analyzed using LCN and FST.” Letter dated Sept. 7, 2024 at 2. Unlike in Defendants' cited cases, here, OCME used LCN and FST testing on a non-deducible DNA sample that was a mere 22.35 pg in quantity - which translates to “a little over three cells” of human DNA. O'Connor Deposition at 169.
For context, the human body is made of many trillions of cells, and the DNA found in “typical fingerprints contain on the order of 100 cells.” Krane Report at 56.
Given this specific combination of factors, each of which can dramatically impact OCME's interpretation of a DNA sample, the Court must be stalwart in exercising its gatekeeping function and “make sure that the highest standards of reliability are maintained.” Williams, 35 N.Y.3d at 29. Because those standards were not met here, the Court grants Plaintiff's motion to preclude.
CONCLUSION
For the foregoing reasons, the Court grants Plaintiff's motion to preclude Defendants from offering the DNA evidence generated by OCME in this case.
SO ORDERED.