SRI International

Patent Trials and Appeals Board

Apr 1, 2021

2019006267 (P.T.A.B. Apr. 1, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
14/332,071 07/15/2014 Hui Cheng SRI7093-2 5832
14824 7590 04/01/2021
Moser Taboada/ SRI International
1030 Broad Street
Suite 203
Shrewsbury, NJ 07702
EXAMINER
SHELEHEDA, JAMES R
ART UNIT PAPER NUMBER
2424
NOTIFICATION DATE DELIVERY MODE
04/01/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
docketing@mtiplaw.com
llinardakis@mtiplaw.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________

Ex parte HUI CHENG, HARPREET SINGH SAWHNEY, JINGEN LIU,
ISHANI CHAKRABORTY, OMAR JAVED, and STEVEN S. WEINER
___________

Appeal 2019-006267
Application 14/332,071
Technology Center 2400
____________

Before JAMES B. ARPIN, IRVIN E. BRANCH, and
PHILLIP A. BENNETT, Administrative Patent Judges.

ARPIN, Administrative Patent Judge.

DECISION ON APPEAL
Appellant1 appeals under 35 U.S.C. § 134(a) from the Examiner’s
decision rejecting claims 1–27, all of the pending claims. Appeal Br. 1.2
We have jurisdiction under 35 U.S.C. § 6(b).
We affirm.

1 “Appellant” refers to “applicant” as defined in 37 C.F.R. § 1.42.
Appellant identifies the real party-in-interest as SRI International. Appeal
Br. 3.
2 In this Decision, we refer to Appellant’s Appeal Brief (“Appeal Br.,” filed
April 8, 2019) and Reply Brief (“Reply Br.,” filed August 26, 2019); the
Final Office Action (“Final Act.,” mailed October 2, 2018), Advisory Action
(“Adv. Act.,” mailed December 12, 2018), and the Examiner’s Answer
(“Ans.,” mailed June 25, 2019); and the Specification (“Spec.,” filed July 15,
2014). Rather than repeat the Examiner’s findings and Appellant’s
contentions in their entirety, we refer to these documents.
Appeal 2019-006267
Application 14/332,071

2

STATEMENT OF THE CASE
The claimed systems and machine accessible storage media relate to
“identifying, describing, and sharing salient events depicted in images and
videos [and for] execut[ing] feature detection algorithms on multimedia
input (e.g., video and/or images).” Spec., Abstract. Appellant’s Figure 5
depicts limitations of the claimed systems and media with respect to a
birthday event, and Figure 5 is reproduced below.

The Specification discloses:
Referring now to FIG. 5, an example 500 of salient event
segment identification as disclosed herein is shown. Video 510
is an input to the system 100. The system 100 analyzes the video
using feature detection algorithms and semantic reasoning as
described above. Time dependent output of the system 100’s
semantic analysis of the detected features is shown by the
Appeal 2019-006267
Application 14/332,071

3

graphics 512, 514, 516, 518. In the graphic 512, the portion 520
represents a salient event segment 112 of the video 510. The
system 100 has determined, using the techniques described
above, that the salient event segment 520 depicts the salient
activity of blowing out candles. Similarly, the system 100 has
identified, using the techniques described above, salient event
segments 522, 524, each of which depicts a person singing, and
a salient event segment 526, which depicts a person opening a
box. The system 100 did not detect any segments depicting the
activity of cutting a cake, in the video 510. The system 100 can
extract the segments 520, 522, 524, 526 from the video 510 and
incorporate the segments 520, 522 524, 526 into a video clip that
includes only the most interesting or salient portions of the
video 510.
Spec. ¶ 59.
As noted above, claims 1–27 are pending. Claims 1, 10, and 20 are
independent. Appeal Br. 17 (claim 1), 19–20 (claim 10), 22–23 (claim 20)
(Claims App.). Claims 2–9 depend directly or indirectly from claim 1,
claims 11–19 depend directly or indirectly from claim 10, and claims 21–27
depend directly or indirectly from claim 20. Id. at 17–25.
Claim 1, reproduced below with disputed limitations emphasized, is
representative.
1. A non-transitory machine accessible storage
medium comprising instructions executable by one or more
processors to cause the computing system to:
detect a plurality of different features in a plurality of
different segments of a video by executing a plurality of different
static and dynamic feature detection algorithms on the video
including but not limited to at least one of a Scale-Invariant
Feature Transform, a Spatio-Temporal Interest Point, a Dense
Trajectory based Histograms of Oriented Gradients, or a Dense-
Trajectory based Motion Boundary Histogram, each of the video
segments comprising one or more frames of the video;
Appeal 2019-006267
Application 14/332,071

4

automatically determine an event evidenced by the
detected features using semantic reasoning techniques;
determine a plurality of salient activities having been
known to occur as part of the event by comparing the
automatically determined event to a knowledge base associating
events with respective salient activities known to occur as part of
the event; and
select, based on the determined plurality of salient
activities, at least one feature detection algorithm to identify a
plurality of different salient event segments of the video depicting
at least one of the determined plurality of salient activities.
Id. at 17 (emphases added).
REFERENCES AND REJECTIONS
The Examiner relies upon the following references:
Name3 Reference Published Filed
Ghanem Context-Aware
Learning for Automatic
Sports Highlight
Recognition, 21st
International
Conference on Pattern
Recognition (ICPR)
Pages: 1977–804
Nov. 11–15, 2012 NA
Curcio US 2014/0161354 A1 June 12, 2014 Dec. 6, 2012
Boiman US 2015/0302894 A1 Oct. 22, 2015 June 28, 2015
Mate US 2016/0012293 A1 Jan. 14, 2016 July 8, 2014

3 All reference citations are to the first named author or inventor only.
4 See Final Act. 13 (providing Ghanem citation).
Appeal 2019-006267
Application 14/332,071

5

The Examiner rejects:
1. claims 1–7, 10–16, and 18–25 under 35 U.S.C. § 103 as obvious over
the combined teachings of Curcio and Mate (Final Act. 3–13);
2. claims 8 and 17 under 35 U.S.C. § 103 as obvious over the combined
teachings of Curcio, Mate, and Ghanem (id. at 13–15); and
3. claims 9, 26, and 27 under 35 U.S.C. § 103 as obvious over the
combined teachings of Curcio, Mate, and Boiman (id. at 15–17).
We review the appealed rejections for error based upon the issues
identified by Appellant, and in light of the contentions and evidence
produced thereon. Ex parte Frye, 94 USPQ2d 1072, 1075 (BPAI 2010)
(precedential). The Examiner and Appellant focus their findings and
contentions on claim 1; so do we. See Appeal Br. 12; Ans. 6; Reply Br. 12.
Arguments not made are forfeited.5 Unless otherwise indicated, we adopt
the Examiner’s findings in the Final Office Action and the Answer as our
own and add any additional findings of fact for emphasis. We address the
rejections below.
ANALYSIS
A. Obviousness over Curcio and Mate
As noted above, the Examiner rejects claims 1–7, 10–16, and 18–25
as obvious over the combined teachings of Curcio and Mate. Final Act. 3–

5 See In re Google Tech. Holdings LLC, 980 F.3d 858, 862 (Fed. Cir. 2020)
(“We interpret the Patent Office to be arguing that Google’s failure to raise
its lexicography arguments, inadvertent or not, compels a finding of
forfeiture.”); 37 C.F.R. § 41.37(c)(1)(iv) (“Except as provided for in
§§ 41.41, 41.47 and 41.52, any arguments or authorities not included in the
appeal brief will be refused consideration by the Board for purposes of the
present appeal.”).
Appeal 2019-006267
Application 14/332,071

6

13. Appellant contests only the rejection of claim 1 separately. Appeal
Br. 7–12; Reply Br. 6–12. For the reasons given below, we are not
persuaded the Examiner errs, and we sustain this rejection.
With respect to claim 1, the Examiner finds that Curcio teaches or
suggests the majority of the recited limitations. Final Act. 3–4. However, as
noted above, claim 1 recites, “select[ing], based on the determined plurality
of salient activities, at least one feature detection algorithm to identify a
plurality of different salient event segments of the video depicting at least
one of the determined plurality of salient activities.” Appeal Br. 17 (Claims
App.). The Examiner finds the combined teachings of Curcio and Mate
teach or suggest this limitation, but Appellant disagrees.
Although the Examiner finds Curcio teaches or suggests,
identifying a plurality of different salient event segments of the
video depicting at least one of the determined plurality of salient
activities (identifying salient events segments indicated within
the particular event template; paragraph 81-83, 119-124),
[Curcio] fails to specifically disclose selecting at least one
feature detection algorithm based on the determined plurality of
salient activities.
Final Act. 4 (emphasis added); see Ans. 4. Thus, the Examiner finds Curcio
teaches or suggests the second portion of this disputed limitation, but not the
first portion.
Nevertheless, “Mate was then relied upon for selecting a particular
feature detection algorithm based on the determined plurality of salient
activities (utilizing the particular detection algorithm for the desired salient
object associated with the salient activity: paragraph[s] 35-36, 49-51).” Id.
Specifically, Mate discloses, “[a] salient object is an object that is associated
with a salient event that is to be identified.” Mate ¶ 35. Mate explains:
Appeal 2019-006267
Application 14/332,071

7

In the context of a sporting event having goals positioned at
opposite ends of the field or court and in which the salient event
is the scoring of a goal by either of the teams, the goals may be
the salient objects as any scoring will be accomplished in
conjunction with the goals.
Id. ¶ 36; see Curcio ¶ 121 (“For example, if ‘soccer’ sport type is detected,
only those salient events related to soccer are considered in successive
analysis steps, such as the ‘soccer-goal scoring’ event, the ‘corner kick’
event, ‘penalty kick’ event, ‘free kick’ event, etc., whereas events related to
other sport[s] (such as the ‘start’ event in swimming competitions) are not
considered.”). Thus, referring to Appellant’s claim 1, Mate’s and Curcio’s
“salient events” correspond to the recited “activities” associated with a
recited “event.”
Mate further discloses:
For example, in an instance in which the salient object is a
basketball goal, the apparatus 10, such as the processor, the
memory 14 or the like, may include a predefined definition of the
basketball hoop that is utilized by the detection algorithm during
image recognition in order to identify the image(s) of the salient
video segment that includes the salient object.
Mate ¶ 49 (emphasis added). The detection algorithm may be selected from
a library of algorithms. Id.
Appellant contends the Examiner errs in rejecting claim 1 for three
reasons.6 Appeal Br. 7–12; Reply Br. 6–12. After considering Appellant’s
contentions, we are not persuaded the Examiner errs.

6 In its Reply Brief, Appellant also contends that “Mate is not prior art and
specifically is not a proper 35 U.S.C. §103 reference and cannot be cited as
prior art against the Appellants’ present application.” Reply Br. 2–6; but see
Adv. Act. 2 (rejecting Appellant’s priority claim). Nevertheless, Appellant
did not present this contention in the Appeal Brief, and the Examiner did not
Appeal 2019-006267
Application 14/332,071

8

First, Appellant contends that, unlike claim 1, Mate discloses use of a
“salient object,” e.g., a basketball hoop (see Mate, Fig. 3), in conjunction
with a detection algorithm to identify video segments associated with
“salient events,” i.e., event activities. Appeal Br. 9–10; Reply Br. 7–9.
Specifically, Appellant contends,
Mate teaches that an apparatus, such as a processor, a memory or
the like, may include a predefined definition of the salient object
that is utilized by a detection algorithm during image recognition
in order to identify the image(s) of the salient video segment that
includes the salient object to define regions of interest and that,
after a salient object has been identified, that a salient event is
identified by analyzing only the images of the spatio-temporal
regions of interest using program code instructions such that
other images captured by the image capturing device need not
be analyzed in order to identify the salient events.
Appeal B. 10 (emphases added). Thus, Appellant contends that Mate may
use more information, e.g., salient objects, when utilizing a detection
algorithm to identify salient video segments.
The Examiner acknowledges that
Mate discloses wherein the determination of the salient object
and salient event is based upon the determined subject and type
of the video, as a sporting event will have a salient events
regarding a team scoring and a concert will have salient events
regarding a particular performer (paragraph 35-36). Thus, within

raise this issue in the Answer. Appellant may not properly present this
contention for the first time in the Reply Brief, because the Examiner has not
been given an opportunity to respond. Nor has Appellant identified good
cause for having not presented the issue in its Appeal Brief. Therefore, we
do not consider it here. 37 C.F.R. § 41.41(b)(2) (“Any argument raised in
the reply brief which was not raised in the appeal brief, or is not responsive
to an argument raised in the examiner’s answer . . . , will not be considered
by the Board for purposes of the present appeal, unless good cause is
shown.”); see supra note 4.
Appeal 2019-006267
Application 14/332,071

9

Mate, the selection of the feature detection algorithm to locate a
salient event and salient object is based upon the subject of the
video (paragraph 35-36).
Ans. 5 (emphasis added).
Appellant does not persuade us that claim 1 forecloses consideration
of salient objects when using a detection algorithm “to select, based on the
determined plurality of salient activities, at least one feature detection
algorithm.” See Appeal Br. 17 (Claims App.) (claim 1 recites “comprising
instructions executable to”). Thus, we are not persuaded Mate’s teachings
regarding the consideration of “salient objects” prevents its combination
with Curcio’s teachings to achieve the disputed limitation. Ans. 5; see also
Adv. Act. 2 (citing Curcio ¶ 63 (“The detection module 450 may be
configured to detect a presence of specific objects (e.g., objects of interest)
in a recorded scene.”)). Therefore, we do not find this reason persuasive of
Examiner error.
Second, Appellant contends:
Even if Mate teaches that detection algorithms are selected based
on identified salient objects (which the Appellant does not
concede), in Mate the selected detection algorithms are not
implemented to identify a plurality of different salient event
segments of the video depicting at least one of the determined
plurality of salient activities as claimed by at least the
Appellant’s independent claim 1.
Appeal Br. 10 (emphasis added); see Reply Br. 10–11. Thus, Appellant
contends Mate alone fails to teach or suggest the disputed limitation.
As noted above, however, the Examiner’s findings regarding the
disputed limitation of claim 1 are based on the combined teachings of Curcio
and Mate. Final Act. 4; Ans. 4–5. Appellant cannot show error by attacking
references individually when the rejection is based on a combination of the
Appeal 2019-006267
Application 14/332,071

10

references’ teachings. Ans. 3 (citing In re Merck & Co., Inc., 800 F.2d
1091, 1097 (Fed. Cir. 1986); In re Keller, 642 F.2d 413, 426 (CCPA 1981));
see Adv. Act. 2 (also citing Merck and Keller). Thus, we do not find this
reason persuasive of Examiner error.
Third, Appellant contends,
in contrast to the Appellant’s claims, any allowable combination
of Curcio and Mate only teaches a method and apparatus for
semantic extraction and identifying salient events in which
detected salient events can be used for automatically generating
a video summary of the recorded event and that this can be done
by identifying the video segments containing the salient events
and then stitching such segments one after the other (for example
by preserving the original temporal order) wherein once the final
sport-genre label is obtained, a template is used to identify the
sub-set of salient events which are the most likely to occur in the
videos (as taught in Curcio) in which a predefined definition of
the salient object is utilized by a detection algorithm during
image recognition in order to identify the image(s) of the salient
video segment that includes the salient object to define regions
of interest and a salient event is identified by analyzing only the
images of the defined regions of interest using program code
instructions such that other images captured by the image
capturing device need not be analyzed in order to identify the
salient events (as taught in Mate).
Appeal Br. 11; Reply Br. 11. Nevertheless, Appellant improperly focuses on
the incorporation of Mate’s systems into Curcio’s systems, instead of the
combination of their teachings. See Ans. 5–6; see also Packers Plus Energy
Serv’s Inc. v. Baker Hughes Oilfield Operations, LLC, 773 F. Appx. 1083,
1089 (Fed. Cir. 2019) (“Moreover, obviousness does not require the bodily
incorporation of the teachings of one reference to another reference—an
ordinary artisan has the capacity for ordinary creativity when combining
references.”). As discussed above, we are persuaded the Examiner’s
Appeal 2019-006267
Application 14/332,071

11

combination of the teachings of Curcio and Mate achieves the disputed
limitation of claim 1. Final Act. 3–4; see Adv. Act. 2; Ans. 3–5. Thus, we
do not find this reason persuasive of Examiner error.
On this record, we are not persuaded the Examiner errs in rejecting
claim 1 as obvious over the combined teachings of Curcio and Mate, and we
sustain that rejection. Further, Appellant does not challenge the obviousness
rejection of independent claims 10 and 20, separately from its challenge to
independent claim 1 or of dependent claims 2–7, 11–16, 18, 19, and 21–25,
separately from its challenge to their base claims, independent claims 1, 10,
and 20. Appeal Br. 12; Reply Br. 12; see Ans. 6. Therefore, we also sustain
the rejection of those claims.
B. Obviousness Over Curcio and Mate in Combination with
Ghanem or Boiman
As noted above, the Examiner rejects claims 8, 9, 17, 26, and 27 as
obvious over the combined teachings of Curcio and Mate in combination
with those of Ghanem or Boiman. Final Act. 13–17. Appellant does not
challenge the obviousness rejections of these dependent claims, separately
from the challenge to their base claims, independent claims 1, 10, and 20;
and, therefore, for the reasons given above, we sustain the rejections of
claims 8, 9, 17, 26, and 27. Appeal Br. 12; Reply Br.12; see Ans. 6.
DECISION
1. The Examiner does not err in rejecting:
a. claims 1–7, 10–16, and 18–25 as obvious over the combined
teachings of Curcio and Mate;
b. claims 8 and 17 as obvious over the combined teachings of
Curcio, Mate, and Ghanem; and
Appeal 2019-006267
Application 14/332,071

12

c. claims 9, 26, and 27 as obvious over the combined teachings of
Curcio, Mate, and Boiman.
2. Thus, on this record, claims 1–27 are not patentable.
CONCLUSION
We affirm the Examiner’s rejections of claims 1–27.
In summary:
Claims
Rejected
35 U.S.C.
§
Reference(s)/Basis Affirmed Reversed
1–7, 10–16,
18–25
103 Curcio, Mate 1–7, 10–16,
18–25

8, 17 103 Curcio, Mate, Ghanem 8, 17
9, 26, 27 103 Curcio, Mate, Boiman 9, 26, 27
Overall
Outcome
1–27
No time period for taking any subsequent action in connection with
this appeal may be extended under 37 C.F.R. § 1.136(a). See 37 C.F.R.
§ 1.136(a)(1)(iv).

AFFIRMED