International Business Machines Corporation

Patent Trials and Appeals Board

May 6, 2021

2020000446 (P.T.A.B. May. 6, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/819,127 11/21/2017 Kenneth W. Church AUS820161042US01 1037
40412 7590 05/06/2021
IBM CORPORATION
C/O LESLIE A. VAN LEEUWEN
6123 PEBBLE GARDEN CT.
AUSTIN, TX 78739
EXAMINER
ROBERTS, SHAUN A
ART UNIT PAPER NUMBER
2657
NOTIFICATION DATE DELIVERY MODE
05/06/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
leslie@vl-patents.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
_____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
_____________

Ex parte KENNETH W. CHURCH, DIMITRIOS B. DIMITRIADIS,
PETR FOUSEK, MIROSLAV NOVAK, and GEORGE A. SAON
_____________

Appeal 2020-000446
Application 15/819,127
Technology Center 2600
______________

Before ERIC S. FRAHM, BETH Z. SHAW, and JOYCE CRAIG,
Administrative Patent Judges.

FRAHM, Administrative Patent Judge.

DECISION ON APPEAL

STATEMENT OF THE CASE
Appellant1 appeals under 35 U.S.C. § 134(a) from the Examiner’s
Final Rejection of claims 1–20, all of the claims pending in the instant
application. We have jurisdiction under 35 U.S.C. § 6(b).
We affirm.

1 We use the word “Appellant” to refer to “applicant” as defined in
37 C.F.R. § 1.42. “The word ‘applicant’ when used in this title refers to the
inventor or all of the joint inventors, or to the person applying for a patent as
provided in §§ 1.43, 1.45, or 1.46.” 37 C.F.R. § 1.42(a). Appellant
identifies the real party in interest as International Business Machines
Corporation (Appeal Br. 2).
Appeal 2020-000446
Application 15/819,127

2

DISCLOSED AND CLAIMED INVENTION
Speaker diarization is the process of partitioning an audio stream with
multiple people into homogeneous segments associated with each individual,
is an important part of speech recognition systems. According to Appellant,
“[t]he goal of the diarization systems is to extract information about the
speakers found in an audio document, such as the number of speakers, their
turns, the timing of their turns, etc.” (Spec. ¶ 3).
With this is mind, Appellant discloses and claims a diarization method
that uses a voice activation detection (VAD) process on an audio stream that
contains speech segments from speakers and non-verbal segments, and then
inputs the VAD process output into an automatic speech recognition (ASR)
process to output spoken words corresponding to speech segments that are
devoid of non-verbal segments (see Spec. ¶¶ 4, 7–9; Abstract; claims 1, 8,
15). Claim 1 is exemplary of the subject matter on appeal, and is reproduced
below with emphasis added to disputed portions of the claim:
1. A method implemented by an information handling system
that includes a memory and a processor, the method comprising:
receiving an audio stream that comprises both a plurality
of speech segments corresponding to a plurality of human
speakers and a plurality of non-verbal segments;
utilizing a voice activation detection (VAD) process on the
audio stream, wherein an output of the VAD process is a digital
audio stream of voices corresponding to the plurality of speech
segments;
inputting the VAD process output into an automatic
speech recognition (ASR) process, wherein an output of the ASR
process comprises a plurality of spoken words corresponding to
the plurality of speech segments and is devoid of the plurality of
non-verbal segments;
inputting the ASR process output to a speaker turn
detection (STD) process, wherein the STD process generates a
Appeal 2020-000446
Application 15/819,127

3

plurality of speaker segments that each end at a word boundary
of one of the plurality of spoken words; and
applying a speaker clustering algorithm to the plurality of
speaker segments, wherein the speaker clustering algorithm
associates an identifier of one of the human speakers with each
of the speaker segments.

Appeal Br. 19, Claims Appendix (emphases added).

REJECTIONS
The Examiner made the following rejections:
(1) Claims 1–20 stand rejected under the judicially created doctrine of
non-statutory, obviousness-type double patenting over claims 1–20 of U.S.
Application No. 15/819,158, now Church et al. (US 10,468,031 B2; issued
Nov. 5, 2019) (hereinafter, “Church”)2 (Final Act. 2–6).
(2) Claims 1, 2, 8, 9, and 15 stand rejected under 35 U.S.C. § 103 as
being unpatentable over Ziv et al. (US 2014/0074467 A1; published March
13, 2014) (hereinafter, “Ziv”) (Final Act. 7–9).
(3) Claims 3, 10, and 16 stand rejected under 35 U.S.C. § 103 as being
unpatentable over Ziv and Isensee et al. (US 2016/0253315 A1; published
Sept. 1, 2016) (hereinafter, “Insensee”) (Final Act. 9–10).
(4) Claims 4, 5, 11, 12, 17, and 18 stand rejected under 35 U.S.C.
§ 103 as being unpatentable over Ziv and Chu et al. (US 2016/0027438 A1;
published Jan. 28, 2016) (hereinafter, “Chu”) (Final Act. 10–11).

2 The Examiner’s provisional non-statutory, obviousness-type double
patenting rejection (see Final Act. 2–6) was based on claims 1–20 of U.S.
Application No. 15/819,158, which issued into U.S. Patent No. 10,468,031
on November 5, 2019. Accordingly, the rejection is no longer provisional.
Appeal 2020-000446
Application 15/819,127

4

(5) Claims 6, 7, 13, 14, 19, and 20 stand rejected under 35 U.S.C.
§ 103 as being unpatentable over Ziv, Chu, and Romano et al. (US
2016/0154782 A1; published June 2, 2016) (hereinafter, “Romano”) (Final
Act. 12–15).

ISSUE
Appellant has not presented any arguments as to the merits of the non-
statutory, obviousness-type double patenting rejection over claims 1–20 over
claims 1–20 of Church (see generally Appeal Br. 5–11; Reply Br. 2–8).
Furthermore, we note no terminal disclaimer has been filed or is currently in
the record in the instant application. Therefore, Appellant has not shown
that the Examiner erred in rejecting claims 1–20 under the judicially created
doctrine of non-statutory, obviousness-type double patenting. See 37 C.F.R.
§ 41.37(c)(1)(iv). Because Appellant presents no arguments directed to the
non-statutory, obviousness-type double patenting rejection, this rejection is
summarily affirmed. Manual of Patent Examining Procedure § 1205.02 (9th
ed. Rev. 08.2017, Jan. 2018) (“If a ground of rejection stated by the
examiner is not addressed in the appellant’s brief, appellant has waived any
challenge to that ground of rejection and the Board may summarily sustain
it, unless the examiner subsequently withdrew the rejection in the
examiner’s answer.”); see also 37 C.F.R. § 41.31(c) (“An appeal, when
taken, is presumed to be taken from the rejection of all claims under
rejection unless cancelled by an amendment filed by the applicant and
entered by the Office.”); 37 C.F.R. § 41.37(c)(iv) (“[A]ny arguments or
authorities not included in the appeal brief will be refused consideration by
the Board for purposes of the present appeal.”). Therefore, the rejection of
Appeal 2020-000446
Application 15/819,127

5

claims 1–20 under the judicially created doctrine of non-statutory,
obviousness-type double patenting is sustained, and there is no issue
presented on appeal with regard to this rejection. As a result, we summarily
affirm and do not address this rejection further in our analysis.
Based on Appellant’s arguments in the Appeal Brief (Appeal Br. 5–
11) and the Reply Brief (Reply Br. 2–8), the following principal issue is
presented on appeal:
Did the Examiner err in rejecting claims 1–20 under 35 U.S.C. § 103
as being unpatentable over the base reference Ziv, because there is not
proper motivation and/or reasonable expectation of success to modify Ziv to
teach or suggest the diarization process recited in claims 1, 8, and 15?

ANALYSIS
We have reviewed Appellant’s arguments in the Appeal Brief (App.
Br. 5–11) and the Reply Brief (Reply Br. 2–8) that the Examiner’s
obviousness rejections of claims 1–20 under 35 U.S.C. § 103 as being
unpatentable over the same base reference, Ziv (see Final Act. 7–15) are in
error. We have also reviewed the Examiner’s response to Appellant’s
arguments in the Appeal Brief (Ans. 3–19). For the following reasons, we
agree with Appellant that the Examiner erred in reaching the conclusion of
obviousness over the base reference, Ziv.
Appellant and the Examiner both agree that the difference between
Ziv and the claimed invention is that Ziv’s diarization process is opposite
that of Appellant. Specifically, Appellant’s claimed diarization process uses
a voice activation detection (VAD) process on an audio stream that contains
speech segments from speakers and non-verbal segments, and then inputs
Appeal 2020-000446
Application 15/819,127

6

the VAD process output into an automatic speech recognition (ASR) process
to output spoken words corresponding to speech segments that are devoid of
non-verbal segments (see Spec. ¶¶ 4, 7–9; Abstract; claims 1, 8, 15). In
other words, Appellant inputs the VAD process output into an ASR process.
In contrast, Ziv inputs the ASR process output into the VAD process.
The Examiner determines that it would have been obvious to flip the
order of the VAD and ASR processes to construct Appellant’s invention:
It would have been obvious to one of ordinary skill in the
art at the time of the invention to have it occur/be performed
before the ASR presenting a reasonable expectation of success in
still allowing for the removal of non-speech segments, and
improving [U1e] system in allowing the [U1e] ASR to only focus
on recognizing speech segments.

Final Act. 8 (emphasis added).
Appellant presents arguments (Appeal Br. 5–9; Reply Br. 2–6)
pertaining to, and the dispositive issue in the case before us is, whether the
Examiner has shown there is a reasonable expectation of success in
reversing the ASR and VAD process order of Ziv to achieve the claimed
invention.3

3 Where there is a reason to modify or combine the prior art to achieve the
claimed invention, the claims may be rejected as prima facie obvious
provided there is also a reasonable expectation of success. In re Merck &
Co., Inc., 800 F.2d 1091 (Fed. Cir. 1986); In re Cyclobenzaprine
Hydrochloride Extended-Release Patent Litig., 676 F.3d 1063, 1068–69
(Fed. Cir. 2012) (when arguing a skilled artisan would have been
motivated to combine references, one must show the artisan “would have
had a reasonable expectation of success from doing so.”); see also MPEP
2143.02 (“Reasonable Expectation of Success is Required”), and 2143.02(I)
(“OBVIOUSNESS REQUIRES A REASONABLE EXPECTATION OF
SUCCESS”).
Appeal 2020-000446
Application 15/819,127

7

Reasonable Expectation of Success
The primary basis for a rationale to support a conclusion that a claim
would have been obvious is that all the claimed elements were known in the
prior art and one skilled in the art could have combined the elements as
claimed by known methods with no change in their respective functions, and
the combination would have yielded nothing more than predictable results to
one of ordinary skill in the art. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 538,
416 (2007); Sakraida v. AG Pro, Inc., 425 U.S. 273, 282 (1976);
Anderson’s-Black Rock, Inc. v. Pavement Salvage Co., 396 U.S. 57, 62–63
(1969); Great Atlantic & P. Tea Co. v. Supermarket Equip. Corp., 340 U.S.
147, 152 (1950). However, an invention is not obvious just “because all of
the elements that comprise the invention were known in the prior art;” rather
a finding of obviousness at the time of invention requires a “plausible
rational [sic] as to why the prior art references would have worked
together.” Power-One, 599 F.3d at 1351; Amgen Inc. v. F. Hoffman-LA
Roche Ltd., 580 F.3d 1340, 1362 (Fed. Cir. 2009) (“An obviousness
determination requires that a skilled artisan would have perceived a
reasonable expectation of success in making the invention in light of the
prior art.”).
The reasonable expectation of success requirement refers to the likelihood
of success in combining references to meet the limitations of the claimed
invention. “[F]ailure to consider the appropriate scope of the . . . patent’s claimed
invention in evaluating the reasonable expectation of success . . . constitutes a
legal error.” Allergan, Inc. v. Apotex Inc., 754 F.3d 952, 966 (Fed. Cir. 2014)
(emphasis added). In other words, the correct inquiry is that one must have a
Appeal 2020-000446
Application 15/819,127

8

motivation to combine accompanied by a reasonable expectation of achieving
what is claimed.4
Appellant contends in both the Appeal Brief (Appeal Br. 5–9) and the
Reply Brief (Reply Br. 2–6) that there is no reasonable expectation of
success in switching the order of the processes taught by Ziv so as to render
the pending claims obvious.5 Appellant contends this is due in part to the
fact that the ASR into VAD arrangement claimed is critical (see Reply Br.
5), as supported by the disclosure of multiple benefits to applying the ASR
output to the input of the VAD process in the Specification (see Appeal Br. 8
citing Spec. ¶¶ 8, 9), and thus there is no motivation to rearrange the order
(see Reply Br. 6). We agree.
In this light, the Examiner’s response that “the arrangement is
inconsequential” (Ans. 18) is unreasonable. Similarly, the Examiner’s
determinations that switching the order of the ASR and VAD processes in
the diarization method of Ziv would have been obvious as a simple
rearrangement of parts or as a matter of design choice (see Ans. 9–10), are
also unreasonable.6

4 An invention is obvious if “a skilled artisan would have been motivated to
combine the teachings of the prior art references to achieve the claimed
invention, and that the skilled artisan would have had a reasonable
expectation of success in doing so.” Proctor & Gamble Co. v. Teva Pharm.
USA, Inc., 566 F.3d 989, 994 (Fed. Cir. 2009) (quoting Pfizer, Inc. v.
Apotex, Inc., 480 F.3d 1348, 1361 (Fed.Cir.2007)).
5 See Ex parte Frye, 94 USPQ2d 1072 (BPAI 2010) (precedential) (the
Board reviews the particular finding(s) contested by an appellant anew in
light of all the evidence and argument on that issue).
6 “[T]o have a reasonable expectation of success, one must be motivated to
do more than merely to vary all parameters or try each of numerous possible
Appeal 2020-000446
Application 15/819,127

9

Furthermore, once Appellant has disputed the issue of a lack of
reasonable expectation of success, as has been done in this case, the
Examiner has the burden of showing a reasonable expectation of success
exists (i.e., the Examiner has the burden of production). See, e.g., In re
Nuvasive, No. 15-1670 slip op. at 9 (Fed. Cir. Dec. 7, 2016) (“Our precedent
dictates that the PTAB must make a finding of a motivation to combine
when it is disputed.”). Here, the Examiner has improperly shifted the burden
of showing there is no reasonable expectation of success to Appellant (see
Ans. 18).
In any event, on this record, the Examiner has not met the burden of
showing a reasonable expectation of success in reversing the order of Ziv’s
VAD and ASR processes, and the Examiner’s assertions that (i) “the
proposed rearrangement is very well-known and established in the art,” and
(ii) “putting a VAD before an ASR [as Appellant claims] would have no
reasonable expectation of success would go against current industry research
and norms” (Ans. 16), are conclusory and unsupported by record evidence.
Based on the foregoing, we concur with Appellant’s assertions (see
Appeal Br. 5–9; Reply Br. 2–6) that the Examiner has not shown (i) a
reasonable expectation of success in modifying Ziv to reverse the order of
the ASR and VAD processes; and as a result, (ii) that claims 1–20 are
obvious over the base reference, Ziv.

choices until one possibly arrived at a successful result.” Pfizer, Inc. v.
Apotex, Inc., 480 F.3d 1348, 1365 (Fed. Cir. 2007) (quoting Medichem, S.A.
v. Rolabo, S.L.,437 F.3d 1157, 1165 (Fed. Cir. 2006)); see also In re Stepan
Co., 868 F.3d 1342, 1347 (Fed. Cir. 2017).

Appeal 2020-000446
Application 15/819,127

10

As a result, we find that the Examiner has not properly established
factual determinations and articulated reasoning with a rational underpinning
to support the legal conclusion of obviousness for claims 1–20 on this
record, resulting in a failure to establish prima facie obviousness. Therefore,
we do not sustain the Examiner’s obviousness rejections of claims 1–20.

CONCLUSION
On the record before us, we are constrained to find the Examiner erred
in rejecting claims 1–20 over the base reference, Ziv, because the Examiner
has not sufficiently shown a reasonable expectation of success in modifying
Ziv (by reversing the order of the ASR and VAD processes) to achieve the
inventions recited in claims 1–20 on appeal. Accordingly, we do not sustain
the Examiner’s obviousness rejections based on Ziv.
For the reasons discussed supra in the Issues section, we affirm the
Examiner’s non-statutory, obviousness-type double patenting rejection over
claims 1–20 over claims 1–20 of Church.
Because we have affirmed at least one ground of rejection with
respect to claims 1–20 on appeal, the Examiner’s decision rejecting claims
1–20 is affirmed. See 37 C.F.R. § 41.50(a)(1) (“The affirmance of the
rejection of a claim on any of the grounds specified constitutes a general
affirmance of the decision of the [E]xaminer on that claim, except as to any
ground specifically reversed.”).
In summary:
Claim(s)
Rejected
35 U.S.C. § Reference(s)/Basis Affirmed Reversed
1–20 Non-statutory
Double Patenting
1–20
Appeal 2020-000446
Application 15/819,127

11

1, 2, 8, 9,
15
103 Ziv 1, 2, 8, 9,
15
3, 10, 16 103 Ziv, Isensee 3, 10, 16
4, 5, 11,
12, 17, 18
103 Ziv, Chu 4, 5, 11,
12, 17, 18
6, 7, 13,
14, 19, 20
103 Ziv, Chu, Romano 6, 7, 13,
14, 19, 20
Overall
Outcome
1–20

TIME PERIOD FOR RESPONSE
No time period for taking any subsequent action in connection with
this appeal may be extended under 37 C.F.R. § 1.136(a)(1). See 37 C.F.R.
§§ 41.50(f), 41.52(b).

AFFIRMED