Apple Inc.

Patent Trials and Appeals Board

Jul 29, 2020

14223601 - (D) (P.T.A.B. Jul. 29, 2020)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
14/223,601 03/24/2014 Daniel Kurz P28517USX1
(119-1148USX1)
6569
61947 7590 07/29/2020
Apple - Blank Rome
c/o Blank Rome LLP
717 Texas Avenue, Suite 1400
Houston, TX 77002
EXAMINER
HARRIS, DOROTHY H
ART UNIT PAPER NUMBER
2625
NOTIFICATION DATE DELIVERY MODE
07/29/2020 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
houstonpatents@blankrome.com
mbrininger@blankrome.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________________

Ex parte DANIEL KURZ
____________________

Appeal 2019-002712
Application 14/223,601
Technology Center 2600
____________________

Before JUSTIN BUSCH, BETH Z. SHAW, and JASON M. REPKO,
Administrative Patent Judges.

BUSCH, Administrative Patent Judge.

DECISION ON APPEAL
Pursuant to 35 U.S.C. § 134(a), Appellant1 appeals from the
Examiner’s decision to reject claims 1–4, 6, 14–19, 26–28, 31, 32, and 34–
38, which constitute all the claims pending. We have jurisdiction over the
pending claims under 35 U.S.C. § 6(b).
We reverse.

1 We use the word “Appellant” to refer to “applicant” as defined in
37 C.F.R. § 1.42. Appellant identifies the real party in interest as Apple Inc.
Appeal Br. 3.
Appeal 2019-002712
Application 14/223,601

2
STATEMENT OF THE CASE
RELATED APPEAL
Appellant identifies Appeal Number 2018-004740 (Application No.
15/274,968) as a related proceedings. The Board decided Appeal Number
2018-004740 on September 13, 2019.
INTRODUCTION
The invention generally relates to touch detection between two objects
using thermal images. Spec. ¶¶ 2, 14, 15, Abstract. More specifically, the
claimed subject matter relates to obtaining a thermal image of an object,
identifying multiple temperature intervals (a first temperature associated
with a first object, a second temperature associated with a second object, and
a third temperature between the first and second temperatures), and detecting
a touch between the first and second objects based on a number of pixels
having temperatures in the third interval being greater than a threshold
number. Spec. ¶¶ 33, 50, 76, 79, 94, Fig. 4. Claims 1, 26, and 27 are
independent claims. Claim 1 is reproduced below:
1. A method of detecting a touch between at least part
of a first object and at least part of a second object, wherein the
first object has a different temperature than the second object,
comprising:
receiving, from a thermal camera, a thermal image of a
portion of a first object and a portion of a second object;
identifying, based on the thermal image, a first
temperature interval associated with the first object and a second
temperature interval associated with the second object;
identifying a third temperature interval between the first
temperature interval and the second temperature interval; and
detecting a touch between the first object and the second
object in response to detecting that a number of pixels in a group
of pixels in the thermal image satisfies a size threshold, each
Appeal 2019-002712
Application 14/223,601

3
pixel in the group of pixels having a temperature value within the
third temperature interval.

THE PENDING REJECTIONS
Claims 1–4, 18, 27, and 36–38 stand rejected under 35 U.S.C. § 103
as obvious in view of Elliot N. Saba et al., Dante Vision: In-Air and Touch
Gesture Sensing for Natural Surface Interaction with Combined Depth and
Thermal Cameras, ESPA 2012 (2012); Eric Larson et al., HeatWave:
Thermal Imaging for Surface User Interaction, CHI 2011, 2565–2574 (May
7, 2011); and Nobuyuki Otsu, A Threshold Selection Method from Gray-
Level Histograms, IEEE Trans. on Sys., Man, and Cybernetics vol. SMC-9
No. 1 (Jan. 1979), 62–66. Final Act. 4–13.
Claims 6, 14–17, 19, 26,2 28, 31, and 35 stand rejected under
35 U.S.C. § 103 as obvious in view of Saba, Larson, Otsu, Junuzovic (US
2012/0320158 A1; Dec. 20, 2012), and Stafford (US 2013/0257751 A1;
Oct. 3, 2013). Final Act. 13–27.
Claims 32 and 34 stand rejected under 35 U.S.C. § 103 as obvious in
view of Saba, Larson, Otsu, Junuzovic, Stafford, and Parshionikar (US
2014/0078049 A1; Mar. 20, 2014). Final Act. 27–29.

2 The Examiner finds Stafford teaches a medium comprising code, but we
cannot discern which, if any, of claim 26’s limitations the Examiner finds
Junuzovic teaches or suggests. Final Act. 22–24 (citing Stafford ¶ 122).
Similarly, we cannot discern which, if any, limitations in claims 15, 19, 28,
and 31 the Examiner finds Junuzovic teaches or suggests. Final Act. 19–22,
24–25.0
Appeal 2019-002712
Application 14/223,601

4
ANALYSIS
The Examiner finds the combination of Saba, Larson, and Otsu
teaches or suggests every limitation recited in independent claims 1 and 27,
and the Examiner finds the combination of Saba, Larson, Otsu, Junuzovic,
and Stafford teaches or suggests every limitation recited in independent
claim 26. Final Act. 4–11, 22–24. Appellant argues that the combination of
Saba, Larson, and Otsu fail to teach or suggest “detecting a touch between
the first object and the second object in response to detecting that a number
of pixels in a group of pixels in the thermal image satisfies a size threshold,
each pixel in the group of pixels having a temperature value within the third
temperature interval” (the “detecting a touch” step), as recited in
independent claim 1 and commensurately recited in independent claims 26
and 27. Appeal Br. 6–9. Appellant further argues the additionally cited
references— Junuzovic, Stafford, and Parshionikar —do not cure this
deficiency. Appeal Br. 8–10.
Saba describes a system that senses gestures and touches with respect
to natural (i.e., “un-instrumented”) surface interactions using both depth and
thermal imaging. Saba, Abstract. Saba teaches determining that a frame
contains only background if the image contains no large connected heat
sources for ten continuous frames. Saba ¶ 3.2.2. On the other hand, if Saba
detects a connected heat source occupying a large area, Saba (1) isolates
objects above the surface using a depth image, which is calculated by
subtracting the background depth image from the current depth image and
(2) uses Otsu’s thresholding method to segment the current thermal image.
Saba § 3.2.2. Saba discloses that Otsu’s method is efficient to “dynamically
track[] the optimal point of separation in the temperature histogram”
Appeal 2019-002712
Application 14/223,601

5
“[b]ecause the background and hand temperatures can be assumed to occupy
significantly different temperature ranges.” Saba § 3.2.2; see Otsu 66.
Saba determines the center of the hand using the segmented thermal
image, then transforms the “centroid of the hand . . . into depth space (using
the homography), resulting in an estimate of the height of the hand over the
surface.” Saba § 3.2.3. Using the depth image, Saba extracts “mean and
median finger point depth around a neighborhood of 5x5 pixels” and feeds
the features into a tree classifier, which outputs the state as either “touch
down” or “hover.” By saving previous depth frames, Saba can retrain the
touch down classifier by applying Larson’s residual heat transfer detection
method to corresponding previous frames from the thermal camera. Saba
§ 3.2.5.
Larson describes “a system that uses digital thermal imaging cameras
to detect, track, and support user interaction on arbitrary surfaces.” Larson,
Abstract. Larson focuses on software to detect and extract heat traces,
which “are the residual heat left behind on a surface due to the heating of
that surface by another warmer object, such as a human hand.” Larson
2568. Larson looks for heat traces only “in pixel locations where the hand
has recently traveled.” Larson, 2568. Larson calibrates its system by
dynamically computing a background image whenever no human hand is
present in the image. Larson 2568. Larson, like Saba, segments thermal
images “on a real-time, frame by frame basis using well known thresholding
methods, such as Otsu’s method.” Larson 2566–67; see also id. at 2568–69
(explaining that Otsu’s method, which uses “the gray-level histogram . . . to
maximize the separation of pixel gray levels between two classes,” “is ideal
Appeal 2019-002712
Application 14/223,601

6
in thermal sensing because a hand’s temperature is almost always distinct
from the background”).
In discussing prior work in the field, Larson discloses that, by using
thermal imaging, “one can use . . . the size and heat of the touch spot left
behind to infer the pressure a user exerts on a surface.” Larson 2567
(emphasis added). To do so, Larson trains the system to identify various
pressures and classifies each heat trace using a tree classifier based on the
detected pressure of each trace. Larson 2570.
The Examiner finds Saba teaches connected heat sources occupying a
large area correspond to a hand and, when such a large connected heat
source is detected, Saba uses Otsu’s method to segment portions of the
thermal image having different temperatures. Ans. 3 (citing Saba § 3.2.2);
Final Act. 5–6. The Examiner further finds Saba teaches using Larson’s
residual heat transfer detection to retrain the touch classifier and explicitly
discloses that the system knows the user was pressing down on the surface if
it detects residual heat transfer in previous thermal images. Ans. 3 (citing
Saba § 3.2.5); Final Act. 6. The Examiner also finds Larson teaches using
heat traces to accurately distinguish between hovering and Larson further
teaches using the size and heat of a touch spot to infer the pressure a user
exerts on the surface. Ans. 3–4 (citing Larson 2568–70); Final Act. 6. The
Examiner finds Larson teaches looking for heat traces in pixel locations
where a hand has traveled recently and evaluating heat transfer pixels to
determine whether to classify each pixel as a heat trace. Ans. 4 (citing
Larson 2569 (“Uncalibrated Heat Trace Detection”). The Examiner then
concludes that the combination of Saba, Larson, and Otsu’s teaches or
Appeal 2019-002712
Application 14/223,601

7
suggests the detecting a touch step. Ans. 4–5 (citing Saba §§ 3.2.2, 3.2.5;
Larson 2569 (“Uncalibrated Heat Trace Detection”).
Appellant argues that Saba fails to teach detecting a touch in response
to detecting that a size of a group of pixels within a particular temperature
interval exceed a threshold because Saba detects touches on a surface based
on the depth image. Appeal Br. 6. Appellant acknowledges that Larson
teaches using a size of a group of pixels to determine the amount of pressure
exerted in an area that already has been determined to be a touch. Reply
Br. 2. However, Appellant contends Larson detects a touch without
considering a size of a group of pixels and therefore fails to teach detecting a
touch based on (or “in response to”) a size of a group of pixels. Reply Br. 2
(arguing “Larson determines the likelihood of each pixel being part of a heat
trace based on a temporally smoothed temperature, a temporal derivative,”
and “Larson uses the size of the group of pixels to detect an attribute of the
already detected touch, but does not use the size of the group of pixels to
detect that the touch has occurred”); Appeal Br. 6–7.
When construing claim terminology during prosecution before the
Office, claims are to be given their broadest reasonable interpretation
consistent with the Specification, reading claim language in light of the
Specification as it would be interpreted by one of ordinary skill in the art. In
re Am. Acad. of Sci. Tech Ctr., 367 F.3d 1359, 1364 (Fed. Cir. 2004).
However, the broadest reasonable interpretation differs from the
broadest possible interpretation. In re Smith Int’l, Inc., 871 F.3d 1375, 1383
(Fed. Cir. 2017). The correct inquiry in giving a claim term its broadest
reasonable interpretation in light of the specification is “an interpretation
that corresponds with what and how the inventor describes his invention in
Appeal 2019-002712
Application 14/223,601

8
the specification, i.e., an interpretation that is ‘consistent with the
specification.”’ Smith, 871 F.3d at 1382–83 (quoting In re Morris, 127 F.3d
1048, 1054 (Fed. Cir. 1997)). We presume that claim terms have their
ordinary and customary meaning. See In re Translogic Tech., Inc., 504 F.3d
1249, 1257 (Fed. Cir. 2007) (“The ordinary and customary meaning is the
meaning that the term would have to a person of ordinary skill in the art in
question.”) (internal quotation marks omitted).
Appellant’s Specification provides examples in which touch detection
is based on the size of a “cluster” or “blob” in the thermal image that meets
certain constraints. See, e.g., Spec. ¶¶ 76, 79, 93, 94, Fig. 4. In particular,
paragraph 76 discloses constraining the cluster size “to consider as resulting
from a touch.” Spec. ¶ 76. In the provided example, a cluster of pixels in
the intermediate temperature range (i.e., a temperature range between the
temperatures of the two objects that potentially touched) are only considered
to be the result of a touch if the radios is at least 5 pixels and not greater than
50 pixels. Spec. ¶ 76; see also Spec. ¶¶ 79 (“If these clusters satisfy certain
constraints, e.g. on their minimal size, or average temperature, a touch is
considered to be detected.”), 94 (“The actual touch can then be detected for
example by means of blob detection . . . considering only blobs with a
temperature falling into the touch interval 408 and having a reasonable
size.”), Fig. 4 (depicting three temperature ranges and the number of pixels
within those temperature ranges). Accordingly, we construe the detecting a
touch step to require detecting that the first and second object touched in
response to detecting that a certain number (above a minimum constraint
and/or below a maximum constraint) of pixels fall within a temperature
Appeal 2019-002712
Application 14/223,601

9
interval between the temperature interval associated with the first object and
the temperature interval associated with the second object.
We are persuaded by Appellant’s arguments. We agree with the
Examiner’s findings regarding the individual teachings cited in Saba,
Larson, and Otsu. In particular, we agree with the Examiner that Larson
teaches (1) using heat traces to differentiate between a hover interaction and
a touch interaction between two objects and (2) determining the exerted
pressure of a touch based on the number and heat of pixels in the detected
touch. See Ans. 3–4 (citing Larson 2568–70); Final Act. 6. We also agree
with the Examiner that Saba explicitly retrains its touch classifier using
Larson’s residual heat trace detection method. Saba § 3.2.5; see Ans. 3;
Final Act. 6.
However, as Appellant argues, Larson teaches using the number of
pixels in an already-detected touch to determine the pressure exerted by a
first object on the surface of a second object. See Reply Br. 2; Appeal Br. 6–
7; accord Ans. 3–4; Final Act. 6. The Examiner has not sufficiently
explained or supported how or why a person of ordinary skill in the art
would have modified Saba’s or Larson’s touch detection method to use
Larson’s method for determining the pressure exerted for a given touch
using the number of pixels in the detected touch.
For the reasons discussed above, on this record we reverse the
rejection of independent claim 1; independent claim 27, which recites a
commensurate limitation; and claims 2–4, 18, and 36–38, which depend
directly from claim 1 and incorporate claim 1’s limitations via dependency
as obvious in view of Saba, Larson, and Otsu. We also reverse the rejection
of independent claim 26 as obvious in view of Saba, Larson, Otsu,
Appeal 2019-002712
Application 14/223,601

10
Junuzovic, and Stafford for the same reasons because claim 26 recites a
commensurate limitation and the Examiner does not find Junuzovic or
Stafford teach or suggest the detecting a touch step. For the same reasons,
we reverse the rejection of claims 6, 14–17, 19, 28, 31, and 35, which
ultimately depend from one of claims 1 and 27 and incorporate the
respective limitations, as obvious in view of Saba, Larson, Otsu, Junuzovic,
and Stafford. We also reverse the rejection of claims 32 and 34, which
ultimately depend from claim 1 and incorporate the limitations of claim 1, as
obvious in view of Saba, Larson, Otsu, Junuzovic, Stafford, and
Parshionikar because the Examiner does not find Parshionikar teaches or
suggests the detecting a touch step.
DECISION SUMMARY
Claims
Rejected
35 U.S.C. § References Affirmed Reversed
1–4, 18, 27,
36–38
103 Saba, Larson, Otsu 1–4, 18, 27,
36–38
6, 14–17, 19,
26, 28, 31, 35
103 Saba, Larson, Otsu,
Junuzovic, Stafford
6, 14–17, 19,
26, 28, 31, 35
32, 34 103 Saba, Larson, Otsu,
Junuzovic, Stafford,
Parshionikar
32, 34
Overall
Outcome
1–4, 6, 14–
19, 26–28,
31, 32, 34–38

REVERSED