Opinion
Civil Action No. 15-cv-01697-CBS
06-08-2017
ESTES PARK TAFFY COMPANY, LLC, a Colorado limited liability company; BENJAMIN HOLCOMB; and BRET HOLCOMB, Plaintiffs, v. THE ORIGINAL TAFFY SHOP, INC., a Colorado corporation, Defendant.
ORDER REGARDING PLAINTIFF'S MOTION TO EXCLUDE DEFENDANT'S EXPERT REPORT
This matter comes before the court on Plaintiffs/Counterclaim Defendants' Motion to Exclude Defendant's Expert Report (doc. #58). Plaintiffs ask this court to "exclude the report and testimony of Defendant's expert John E. Weiss pursuant to Federal Rules of Evidence, Rules 702, 703 and 403." According to Plaintiffs, Mr. Weiss's report, and presumably the opinions contained therein, do not "meet the Daubert/Kumho tests of relevance and reliability, and as such, should be excluded." Relying upon an "Analysis of John Weiss Report for The Original Taffy Shop" prepared by their own retained expert, Daniel Hoffman, Plaintiffs contend that Defendant's survey is fatally flawed through the use of an over-inclusive target population, biased questions, and various improper survey procedures.
For the uninitiated reader, Plaintiffs Estes Park Taffy Company, LLC, Benjamin Holcomb, and Bret Holcomb brought this action on August 7, 2015, by filing a Complaint (doc. #1) that seeks a declaratory judgment finding that the use of their business name, trade dress, packaging, tagline and mountain design do not constitute trademark or trade dress infringement or unfair competition with regard to Defendants' claimed trademarks and trade dress. Defendant The Original Taffy Shop, Inc. has filed an Answer and Amended Counterclaims (doc. #79) that assert claims under the Lanham Act, the Anti-Cybersquatting Consumer Protection Act, the Colorado Consumer Protection Act, and Colorado common law.
Defendant/Counterclaim Plaintiff's Response to Motion to Exclude Weiss Report (doc. #64) contends that Mr. Weiss was asked "to explore consumer perceptions of any potential customers of a dedicated taffy business, whether aware of the specific dynamics of any given marketplace." To that end, Mr. Weiss designed and conducted a survey "in accordance with generally accepted survey principles." Defendant's Response argues that Mr. Weiss's report and opinions should not be excluded because Plaintiffs suggest nothing more than a disagreement between competing expert "as to the proper choice of methods to apply from a range of generally accepted survey principles."
Plaintiffs filed their Reply to Defendant's Response (doc. #67), reiterating their position that "the Weiss Report/Survey has serious and systemic design flaws that require its exclusion from evidence," and that "Defendant has not met [its] burden to demonstrate that the Weiss Report/Survey is reliable and trustworthy enough to be admitted into evidence."
This court held a hearing on April 11, 2017, during which Mr. Weiss was subjected to extensive direct and cross-examination. The court also accepted, without objection, Plaintiffs' proffer that if Mr. Hoffman were to testify, he would reiterate the information and opinions set forth in his own written reports, and would identify what he perceives to be "core fatal flaws" in Mr. Weiss's methodology and conclusions.
I have reviewed the pending motion and related briefs and exhibits, and the case law cited by the parties. I have also conducted my own research and read the transcript of the hearing on April 11, 2017 (hereinafter "Transcript of Proceedings" (doc. #82)). For the following reasons, I am denying Plaintiffs' motion.
In addition to pertinent case law, I have reviewed (and found particularly informative) Shari Seidman Diamond, J.D., Ph.D.'s Reference Guide on Survey Research (hereinafter "Professor Diamond's Reference Guide"), included in the REFERENCE MANUAL ON SCIENTIFIC EVIDENCE at 359-423 (3d ed. 2011) published by the Federal Judicial Center and the National Research Council of the National Academies and available at https://www.fjc.gov/content/reference-manual-scientific-third-edition-1.
ANALYSIS
Rule 702 of the Federal Rules of Evidence provides that "[i]f scientific, technical or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training or education, may testify thereto in the form of an opinion or otherwise." The rule imposes three requirements for the admission of expert testimony. First, the expert must be qualified by specialized knowledge, skill, experience, training or education to testify on the subject matter of his or her testimony. Second, the testimony must be "'based upon sufficient facts or data,' 'the product of reliable principles and methods,' and the product of the reliable application of these principles and methods to the facts of the case." Cook v. Rockwell International Corp., 580 F. Supp. 2d 1071, 1082 (D. Colo. 2006). Finally, the proffered expert testimony must be relevant to an issue in the case and assist the jury in its deliberations. Ultimately, the proponent of expert testimony has the burden of establishing the admissibility of the expert's opinions under Rule 702. In re Breast Implant Litig., 11 F. Supp. 2d 1217, 1222 (D. Colo. 1998).
Rule 702 does not impose an "overly rigorous" requirement of expertise. The Tenth Circuit has acknowledged that "[a]s long as an expert stays 'within the reasonable confines of his subject area,' our case law establishes 'a lack of specialization does not affect the admissibility of [the expert] opinion, but only its weight." Ralston v. Smith & Nephew Richards, Inc., 275 F.3d 965, 969 (10th Cir. 2001) (quoting Wheeler v. John Deere Co., 935 F.2d 1090, 1100 (10th Cir. 1991)). In this case, the parties have stipulated that Mr. Weiss is qualified to offer expert opinions in the areas of marketing and survey research. See Transcript of Proceedings, at 7.
The district court performs an important gatekeeping function in assuring that each of the foregoing prerequisites is satisfied. Macsenti v. Becker, 237 F.3d 1223, 1230-34 (10th Cir. 2001). However, in discharging its gatekeeper responsibilities, the court remains mindful that "Rule 702 mandates a liberal standard" for the admissibility of expert testimony. Cook, 580 F. Supp. 2d at 1082. The decision to admit or exclude expert testimony is reviewed for abuse of discretion. Etherton v. Owners Ins. Co., 829 F.3d 1209, 1216 (10th Cir. 2016) (noting that an appellate court "must afford substantial deference to the district court's application of Daubert") (emphasis in original). Cf. Valente v. Oak Leaf Outdoors, Inc., No. 14-12892, 2015 WL 4488561, at *2 (E.D. Mich. Jul. 23, 2015) ("Courts take a flexible approach to deciding Rule 702 motions and have broad discretion in determining whether to admit or exclude expert testimony.").
Although an expert witness is permitted wide latitude to offer opinions, including those not based on first-hand knowledge or observation, the opinions must have a reliable basis. Under Rule 702, admissible expert testimony must be based on "actual knowledge and not 'subjective belief or unsupported speculation.'" Mitchell v. Gencorp Inc., 165 F.3d 778, 780 (10th Cir. 1999). In addressing the principles enunciated in Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579 (1993), the Tenth Circuit has recognized that proposed expert testimony must be supported by "appropriate validation - i.e., 'good grounds,' based on what is known."
The plaintiff need not prove that the expert is undisputably correct or that the
expert's theory is "generally accepted" in the scientific community. Instead, the plaintiff must show that the method employed by the expert in reaching the conclusion is scientifically sound and that the opinion is based on facts which sufficiently satisfy Rule 702's reliability requirements.Mitchell, 165 F.3d at 781 (internal citations omitted). A trial court has considerable discretion in determining whether particular expert testimony is reliable. Kumho Tire Co., Ltd. v. Carmichael, 526 U.S. 137, 152 (1999).
No single factor should be dispositive in weighing the reliability of an expert's opinions. Ruiz-Troche v. Pepsi Cola of Puerto Rico Bottling Co., 161 F.3d 77, 85 (1st Cir. 1998). Ultimately, the court's inquiry must focus on three areas: (1) whether the expert's testimony is based on sufficient facts or data; (2) whether the expert used reliable principles and methodologies; and (3) whether the expert applied these principles and methods reliably to the facts of the case. Cook, 580 F. Supp 2d at 1085-86. If the court concludes that an expert's testimony satisfies these indicia of reliability, "it is up to the jury to decide whether the expert used the best or most reliable methodology, what weight to accord to his testimony and which of competing experts' opinions should be credited." Id. at 1085.
The trial court may consider, inter alia: (1) whether the opinion is susceptible to testing and has been subjected to such testing; (2) whether the opinion has been subjected to peer review; (3) whether there is a known or potential rate of error associated with the methodology used and whether there are standards controlling the technique's operation; and (4) whether the theory has been accepted in the scientific community. Hollander v. Sandoz Pharms. Corp., 289 F.3d 1193, 1205 (10th Cir. 2002).
The requirement of reliability is "lower than the merits standard of correctness." In re Paoli R.R. Yard PCB Litig., 35 F.3d 717, 744 (3d Cir. 1994). The goal under Rule 702 is reliability, not certainty. "[E]ven if the judge believes there are better grounds for some alternative conclusion, and that there are some flaws in the scientist's methods, if there are good grounds for the expert's conclusion, it should be admitted." Bonner v. ISP Techs., Inc., 259 F.3d 924, 929 (8th Cir. 2001) (quoting Heller v. Shaw Indus., Inc., 167 F.3d 146, 152-53 (3d Cir. 1999)).
As a general rule, the factual basis of an expert opinion goes to the credibility of the testimony, not the admissibility, and it is up to the opposing party to examine the factual basis for the opinion in cross examination. Only if the expert's opinion is so fundamentally unsupported that it can offer no assistance to the jury must such testimony be excluded.Bonner, 259 F.3d at 929-30.
That being said, an expert's opinions should be excluded "where the opinion 'is so fundamentally unsupported that it can offer no assistance to the jury.'" Squires v. Goodwin, 829 F. Supp. 2d 1041, 1053 (D. Colo. 2011) (quoting Hose v. Chicago Nw. Transp. Co., 70 F.3d 968, 974 (9th Cir. 1996)).
While . . . most arguments that "assumptions are unfounded go to the weight, not admissibility, of the testimony, there comes a point at which expert testimony is rightly seen as based on speculation, conjecture, or an assumption so unrealistic as to warrant exclusion through the court's role as gatekeeper.Washington v. Kellwood, Co., 105 F. Supp. 3d 293, 323-24 (S.D.N.Y. 2015) (internal citations omitted). Cf. Etherton, 829 F.3d at 1218 ("Neither Rule 702 nor Daubert 'requires a district court to admit opinion evidence that is connected to existing data only by the ipse dixit of the expert. A court may conclude that there is simply too great an analytical gap between the data and the opinion offered.").
Courts that have addressed consumer surveys in the context of Rule 702 and Daubert have suggested that
The criteria for the trustworthiness of survey evidence are that: (1) the "universe" was properly defined; (2) a representative sample of that universe was selected; (3) the questions to be asked of interviewees were framed in a clear, precise and
non-leading manner; (4) sound interview procedures were followed by competent interviewers who had no knowledge of the litigation or the purpose for which the survey was conducted; (5) the data gathered was accurately reported; (6) the data was analyzed in accordance with accepted statistical principles[;] and (7) objectivity of the entire process was assured.LG Electronics U.S.A., Inc. v. Whirlpool Corp., 661 F. Supp. 2d 940, 952 (N.D. Ill. 2009) (citing Weight Watchers Int'l, Inc. v. Stouffer Corp., 744 F. Supp. 1259, 1272 (S.D.N.Y. 1990) and Shari Seidman Diamond, Reference Guide on Survey Research, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 231 (Federal Judical Ctr. 2d ed 2000)). Cf. University of Kansas v. Sinks, No. 06-cv-2341-JAR, 2008 WL 755065, at *3 (D. Kan. Mar. 19, 2008).
The foregoing factors, however, do not change the fundamental reality that "[t]he probative value of a consumer survey is a highly fact-specific determination and a court may place such weight on survey evidence as it deems appropriate." LG Electronics, 661 F. Supp. 2d at 954 (quoting Johnson & Johnson-Merck Consumer Pharmaceuticals Co. v. Rhone-Poulenc Rorer Pharmaceuticals, Inc., 19 F.3d 125, 134 (3rd Cir. 1994). It remains a generally recognized rule that perceived flaws in an expert's analysis are more properly "tested in the crucible of the adversarial system," and do not provide justification "for truncating that process." U.S. v. 14.38 Acres of Land, 80 F.3d 1074, 1079 (5th Cir. 1996); cf. Borawick v. Shay, 68 F.3d 597, 610 (2d Cir. 1995), cert. denied, 517 U.S. 1229 (1996) ("Daubert reinforces the idea that there should be a presumption of admissibility of evidence" and establishes "a bias in favor of admitting evidence short of that solidly and indisputably proven to be reliable").
Mr. Weiss discussed at length during the hearing on April 11, 2017, the methodology he employed in preparing and conducting his survey. Defendant's expert testified that in preparing his survey he relied upon a well-recognized marketing scale handbook and asked colleagues at Colorado State University's College of Business to critique his methodology, including how questions were framed and ordered, the scale employed in the survey, and the sampling technique. See Transcript of Proceedings, at 14 and 28-29. Mr. Weiss also designed and executed a pre-test that did not identify any anomalies or unusual distributions in the information generated. Id. at 19. Defendant's expert testified that the forum he used to gather actual data, SurveyMonkey, is used extensively to perform marketing and survey research and is considered to be outstanding. Id. at 17. Mr. Weiss testified that in performing his survey, he employed a non-probability sampling technique because he was conducting "a convenience sample" which made it "more important for us to get access to a broad-based population than to protect ourselves from some sort of statistical bias that might occur as a result of how we sample." Id. at 9.
Weiss also conceded that marketing and survey practitioners are not in complete agreement on many topics, such as whether a survey should use a balanced or unbalanced scale, whether a survey should use a forced or unforced response format, and whether a survey should be prefaced by an introduction. Id. at 22, 24-25, and 33-34. Mr. Weiss explained during his testimony how he approached those issues. Mr. Weiss acknowledged that problems could potentially arise as a result of the order (an "ordering effect") in which questions are posed in a survey. To avoid those problems, his survey began with general questions before moving to more specific points. He also included one "negatively scored question" to ascertain whether respondents were simply answering in straight-line manner. Id. at 29-31.
I have carefully reviewed the expert opinions offered by Plaintiffs' expert, Mr. Hoffman. He takes exception to much of the work performed by Mr. Weiss and the opinions derived from the survey he designed and conducted. In particular, Mr. Hoffman believes that the Weiss survey suffers from the following flaws:
But see Great Northern Ins. Co. v. Power Cooling, Inc., No. 06-CV-874 (ERK)(KAM), 2007 WL 4688411, at *11 (E.D.N.Y. Dec. 18, 2007) ("[T]he mere fact that a difference of opinion exists does not make [an expert's] conclusions inherently unreliable. Such differences of opinion and alleged weaknesses in the [expert's] methodologies will go to the weight to be given the expert testimony, not its admissibility.").
1. an over-inclusive universe or target population, which was likely to generate irrelevant opinions from respondents who do not eat taffy, rather than a survey population that "only focused on those who know what taffy is, eat taffy, or would buy it;"
2. a non-probability sample that was predicated on convenience and employed individuals "who like to take surveys," rather than a sample predicated on "a random or systematic selection process in which every participant has a known chance of being selected in advance;"
3. a flawed research design that did not include any pictures of store fronts or videos of taffy pullers and was "unrelated to the shopping dynamics that take place among those who are shopping in Estes Park;"
4. biased question wording that did not include an even number of negatively and positively worded questions, did not rotate questions or possible responses to avoid position bias, and included built-in assumptions that might not have been apparent to respondents;
5. a lack of open-ended questions, which created the potential for bias "and strays considerably from normally accepted survey practices;" andPlaintiffs' counsel addressed the foregoing alleged flaws at length during his cross-examination of Mr. Weiss on April 11, 2017. Although many of those questions raise valid points that may be viewed as significant by a trier of fact, I do not find that Mr. Weiss's survey methodology presents deficiencies so substantial as to render the resulting conclusions and opinions untrustworthy under Rule 702.
6. the use of leading questions that were "worded such that they are easy to respond to in a certain manner" and scaled responses that did not allow options for respondents who were neutral, unsure, or had no opinion.
Professor Diamond notes that "[a] majority of the consumer surveys conducted for Lanham Act litigation present results from nonprobability convenience samples" and the "results of these studies are used by major American companies in making decisions of considerable consequence." She acknowledged, however, that "when respondents are not selected randomly from the relevant population, the expert should be prepared to justify the method used to select respondents." See Professor Diamond's Reference Guide, at 382.
Professor Diamond's Reference Guide notes that while rotating the order of questions and order of response choices is desirable, "it is impossible to know in the abstract whether a particular question suffers much, little, or not at all from an order bias." According to Professor Diamond, "lack of rotation should not preclude reliance on the answer to the question, but it should reduce the weight given to that answer." See Professor Diamond's Reference Guide, at 382.
Professor Diamond acknowledges that phrasing survey questions can be "often difficult." She suggests that "pretest activities may be used to improve the clarity of communications with respondents," and notes that "[t]exts on survey research generally recommend pretests as a way to increase the likelihood that questions are clear and unambiguous." See Professor Diamond's Reference Guide, at 387 and 388. Professor Diamond's Reference Guide acknowledges that "[t]he advantage of open-ended questions is that they give the respondent fewer hints about expected or preferred answers," while "closed-ended questions are more suitable for assessing choices between well-identified opinions or obtaining ratings on a clear set of alternatives." Id. at 392 and 394.
Professor Diamond notes that a survey may use filter questions to reduce guessing by respondents, and that "a survey that uses neither full filters nor quasi-filters may overstate the number of respondents with opinions, if some respondents offering opinions are guessing." However, she also acknowledges that "[r]ecent research on the effects of including a 'don't know' opinion shows that quasi-filters as well as full filters may discourage a respondent who would be able to provide a meaningful answer from expressing it." See Professor Diamond's Reference Guide, at 390-91. --------
It bears repeating that "a flaw in methodology does not automatically disqualify an expert opinion; the flaw must be of such substance as to create a lack of 'good grounds' for the expert's conclusions." Babcock & Wilcox Ebensburg, Power, Inc. v. Zurich American Ins. Co., No. 02- CIVIL-299, 2005 WL 6068838, at *5 and 9 (W.D. Pa. Sept. 8, 2005) ("[w]hether . . . [an] expert might have done a better job is not the test under Rule 702; "[a] judge frequently should find an expert's methodology helpful even when the judge thinks that the expert's technique has flaws sufficient to render the conclusions inaccurate") (quoting In re Paoli, 35 F.3d at 744-45)). Cf. Southland Sod Farms v. Stover Seed Co., 108 F.3d 1134, 1143 (9th Cir. 1997) (rejecting defendants' argument that the opinions offered by plaintiffs' consumer survey expert should have been excluded because his survey was not properly focused on the relevant audience and used leading questions; the appellate court concluded these challenges went only to the weight of the survey, not its admissibility); OraLabs, Inc. v. The Kind Group, LLC, No. 13-cv-00170-PAB-KLM, 2015 WL 4538442, at *5 (D. Colo. Jul. 28, 2015) (in denying plaintiff's motion to exclude testimony from defendants' survey expert, the court concluded that any perceived flaws in the expert's survey universe did not raise a "deficiency that is 'so substantial as to render the survey's conclusions untrustworthy'"); United States v. H & R Block, 831 F. Supp. 2d 27, 34 (D.D.C. 2011) ("The methodological validity of a survey is necessarily a question of degree. . . . [T]echnical deficiencies that can be adequately explored on cross-examination generally go to the weight, rather than the admissibility, of the evidence, unless the methodological deficiencies are so sweeping or fundamental as to render the survey wholly unreliable and therefore inadmissible."); University of Kansas, 2008 WL 755065, at *5 (although the trial court agreed that some of the survey questions at issue were suggestive and leading, and that the survey universe was not properly limited to potential purchasers of defendants' shirts, the "significant flaws in the methodology of this survey . . . go to the weight and not the admissibility of the survey;" these flaws "may be adequately brought to the jury's attention through rigorous cross- examination and the presentation of plaintiff's rebuttal expert," and did not warrant excluding defendant's expert or his opinions); Icon Health & Fitness, Inc. v. Nautilus Group, Inc., No. 1:02CV00109TC, 2005 WL 6015496, at *2 (D. Utah Aug. 29, 2005) (in denying defendant's motion to exclude plaintiff's survey expert's testimony, the court concluded that while the universe of the survey might been more specifically defined, that more safeguards could have been implemented in conducting the survey, and the questions might have been more carefully composed, those alleged flaws did not provide a sufficient basis on which exclude the expert's opinions).
It may well be that Mr. Weiss's methodologies and opinions will not carry the day. It is equally possible that Mr. Hoffman's attacks will fail to persuade a trier of fact. For purposes of today and this ruling, I simply know that in the end, cases should be decided on their merits and Rule 702 should not be applied in a hyper-technical manner that unfairly distorts the jury's fact-finding responsibilities. I am not required to decide whether Mr. Weiss's opinions are correct; rather, my mandate is to determine whether his chosen methodology is reliable and based upon sufficient facts and data. See McCall v. Skyland Grain LLC, No. 08-cv-01128-PAB-BNB, 2010 WL 2543574, at *3 (D. Colo. Jun. 22, 2010). Plaintiffs are certainly free to dispute those opinions and can address through cross-examination any real or perceived shortcomings in Mr. Weiss's analysis. See Daubert, 509 U.S. at 596 ("vigorous cross-examination, presentation of contrary evidence, and careful instruction on the burden of proof are the traditional and appropriate means of attacking shaky but admissible evidence"). Cf. Metavante Corp. v. Emigrant Sav. Bank, 619 F.3d 748, 762 (7th Cir. 2010) (recognizing that while "shaky" expert testimony may be admissible, it is not immune from attack on cross-examination); Southland Sod Farms, 108 F.3d at 1143 n.8 ("Unlike novel scientific theories, a jury should be able to determine whether asserted technical deficiencies undermine a survey's probative value.").
CONCLUSION
Accordingly, for all of the real or perceived flaws in his opinions and methodologies, I find that Mr. Weiss's methodology and analysis, and his associated opinions, have sufficient reliability and relevance to withstand challenge under Rule 702. Plaintiffs' Motion to Exclude Defendant's Expert Report (doc. #58) is denied.
Dated this 8th day of June, 2017.
BY THE COURT:
s/ Craig B. Shaffer
United States Magistrate Judge