Adam G. Kirk et al.

Patent Trials and Appeals Board

Jul 17, 2019

13924464 - (D) (P.T.A.B. Jul. 17, 2019)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
13/924,464 06/21/2013 Adam G. Kirk 339100.01 6601
39254 7590 07/17/2019
Barta, Jones & Foley, P.C.
(Patent Group - Microsoft Corporation)
2805 Dallas Parkway
Suite 222
Plano, TX 75093
EXAMINER
BOYLAN, JAMES T
ART UNIT PAPER NUMBER
2486
NOTIFICATION DATE DELIVERY MODE
07/17/2019 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
docket@bjfip.com
usdocket@microsoft.com
uspto@dockettrak.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________

Ex parte ADAM G. KIRK, CHRISTOPH RHEMANN,
OLIVER A. WHYTE, SHAHRAM IZADI, and
SING BING KANG
___________

Appeal 2018-0085631
Application 13/924,464
Technology Center 2400
____________

Before JASON V. MORGAN, DANIEL N. FISHMAN, and
DAVID J. CUTITTA II, Administrative Patent Judges.

FISHMAN, Administrative Patent Judge.

DECISION ON APPEAL
This is an appeal under 35 U.S.C. § 134(a) of the final rejection of
claims 1 and 4–22, all pending claims.2 Claims 2 and 3 are canceled. App.
Br. Appendix A (A-1). We have jurisdiction under 35 U.S.C. § 6.
We reverse.

1 Appellants assert the real party in interest is Microsoft Technology
Licensing, LLC. Appeal Brief 1.
2 In this Opinion, we refer to the Appeal Brief (“App. Br.,” filed April 30,
2018), the Reply Brief (“Reply Br.,” filed August 27, 2018), the Examiner’s
Answer (“Ans.,” mailed June 26, 2018), the Final Office Action (“Final
Act.,” mailed September 27, 2017), and the original Specification (“Spec.,”
filed June 21, 2013 (as amended November 11, 2015)).
Appeal 2018-008563
Application 13/924,464

2

THE INVENTION
Appellants’ invention relates generally to performing depth estimation
of objects using a stereo pair of images. Spec. ¶ 2. Such depth estimation
techniques find correspondences between small image patches in the pair of
stereo images. Id. To aid in assuring the image patches correspond, “an
active illumination pattern may be applied to the scene” to ensure “that the
patches contain some distinctive texture.” Id. ¶ 4. The active illumination
and stereo cameras may operate in the infrared region of the light spectrum
to render the superimposed texture invisible to human observers of the
scene. Id.
According to Appellants, the depth of pixels in image patches near
discontinuities (e.g., at the boundary between objects in the image patch)
may be incorrectly estimated because the image patches at the discontinuity
“may include pixels from two different depths,” an effect referred to as
“stereo fattening.” Id. ¶ 5. Appellants assert that techniques to generate
Adaptive Support Weights (“ASW”) are known to address this problem by
weighting different parts of the patch. Id. ¶¶ 5–6. In ASW, each pixel of a
patch is assigned a weight based on whether that pixel “is believed to lie at
the same depth as” a pixel of interest p in the patch. Id. ¶ 6. Pixels of the
patch believed to lie at the same depth as pixel p receive a higher weight
than other pixels of the patch. Id.
According to Appellants, ASW techniques rely on an assumption that
pixels of the scene “that have similar depths within a patch generally have
similar colors.” Id. ¶ 7. However, “[t]he problem with computing adaptive
support weights on IR images with active illumination is that the patterned
illumination breaks the assumption that the color of a surface will be
Appeal 2018-008563
Application 13/924,464

3

approximately constant” because “[t]he illumination pattern causes large
intensity / color changes to occur everywhere, not only at object boundaries,
whereby the stereo matching degrades.” Id. ¶ 9.
Appellants’ claimed invention purports to resolve this problem by
“using weights from a non-actively illuminated image to determine patch
similarity for pixel matching in . . . actively illuminated stereo images.” Id.
¶ 11. Appellants’ Figure 6 is reproduced below.

Appeal 2018-008563
Application 13/924,464

4

Figure 6, reproduced above, is a flow diagram representing an
exemplary method for using weights from a non-actively illuminated image
to determine matches of patches in actively illuminated stereo images. Id.
¶ 21. Step 602 captures actively illuminated stereo images and a non-
actively illuminated image. Id. ¶ 45. Step 604 selects a pixel in one
captured image with its surrounding patch of pixels and step 606 selects
patches in the other images that may correspond to the patch in the first
image. Id. Step 608 uses the patch in the non-actively illuminated patch to
determine weights of other pixels in that non-actively illuminated patch and
step 610 uses those determined weights to determine a similarity score for
the corresponding patching in the actively illuminated patches. Id. ¶ 46.
Steps 606 through 610 are repeated until step 612 determines that a
satisfactory similarity score is identified for the pair of actively illuminated
stereo patches at which point step 614 outputs the matching pixels in the
stereo images. Id.
Claims 1, 8, and 20 are the independent claims. Independent method
claim 1, reproduced below, is illustrative:
1. A method comprising:
processing a plurality of images including a first image, a
second image, and a third image, wherein the first image and the
second image are actively illuminated stereo images, and the
third image is a non-actively illuminated image;
determining a first patch in the first image based on a
location of a first pixel of interest, the first patch comprising a
first plurality of pixels surrounding the first pixel of interest;
determining a second patch in the second image based on
a location of a second pixel corresponding to the first pixel, the
second patch comprising a second plurality of pixels surrounding
the second pixel;
Appeal 2018-008563
Application 13/924,464

5

determining a third patch in the third image based on a
location of a third pixel corresponding to at least one of the first
pixel and the second pixel, the third patch comprising a third
plurality of pixels surrounding the third pixel; and
optimizing a similarity score between the first patch and
the second patch in the actively illuminated stereo images by:
determining weights for the third plurality of pixels
in the third patch in the third image based upon a similarity
between the third pixel and respective pixels in the third
plurality of pixels; and
using the weights of the third plurality of pixels in
the third patch to determine the similarity score between
the first patch and the second patch in the actively
illuminated stereo images.
Independent claims 8 and 20 recite similar elements in differing claim
styles.

THE REJECTIONS
Claims 1, 7–9, 15, and 20–22 stand rejected under 35 U.S.C. § 103 as
obvious over the combination of Zheng Gu et al. (Trinocular Disparity
Estimation with Adaptive Support Weight and Disparity Calibration, 47
OPTICAL ENGINEERING, 033603-1–033603-5 (March 2008)) (“Gu1”) and Gu
(U.S. Patent Publication No. 2006/0210146 A1) (“Gu2”). Final Act. 3–15.
Claim 4 stands rejected under 35 U.S.C. § 103 as obvious over Gu1,
Gu2, and Fox et al. (U.S. Patent Publication No. 2006/0238714 A1) (“Fox”).
Final Act. 15–17.
Claims 5, 6, and 17 stand rejected under 35 U.S.C. § 103 as obvious
over Gu1, Gu2, and Motta et al. (U.S. Patent Publication No. 2011/0228097
A1) (“Motta”). Final Act. 18–22.
Appeal 2018-008563
Application 13/924,464

6

Claims 10, 11, and 14 stand rejected under 35 U.S.C. § 103 as obvious
over Gu1, Gu2, and Katz et al. (U.S. Patent Publication No. 2012/0056982
A1) (“Katz”). Final Act. 22–25.
Claim 12 stands rejected under 35 U.S.C. § 103 as obvious over Gu1,
Gu2, and Din-Yuen Chan et al. (Regular Stereo Matching Improvement
System Based on Kinetic-Supporting Mechanism, 3 OPEN JOURNAL OF
APPLIED SCIENCES, 22–26 (2013)) (“Chan”). Final Act. 25–27.
Claim 13 stands rejected under 35 U.S.C. § 103 as obvious over Gu1,
Gu2, Katz, and Tony et al. (U.S. Patent Publication No. 2012/0002045 A1)
(“Tony”). Final Act. 27–29.
Claim 16 stands rejected under 35 U.S.C. § 103 as obvious over Gu1,
Gu2, and Um et al. (U.S. Patent Publication No. 2011/0149031 A1) (“Um”).
Final Act. 29–31.
Claim 18 stands rejected under 35 U.S.C. § 103 as obvious over Gu1,
Gu2, Motta, and Um. Final Act. 31–33.
Claim 19 stands rejected under 35 U.S.C. § 103 as obvious over Gu1,
Gu2, Motta, and Lu et al. (U.S. Patent Publication No. 2010/0289885 A1)
(“Lu”). Final Act. 33–34.

ANALYSIS
Regarding independent claim 1, the Examiner relies on Gu1 for
disclosing processing of three images and determining a corresponding patch
in the three images. Final Act. 4–6. The Examiner also relies on Gu1 for
the “optimizing” step, but finds Gu1 “does not explicitly disclose” that the
first and second images (and the corresponding patches) are actively
illuminated and the third image (and the corresponding patch) is non-
Appeal 2018-008563
Application 13/924,464

7

actively illuminated. Id. at 6–8. The Examiner finds Gu2, in the same field
of endeavor, discloses these features in combination with Gu1. Id. at 8–9.
The Examiner further finds:
It would have been obvious to the person of ordinary skill
in the art at the time of the invention to modify the system by
Gu 1 to add the teachings of Gu 2, in order to improve the stereo
matching response time by avoiding a need to switch on/off the
pattern being projected from a projector for capturing the image
with the superimposed features (i.e. active illuminated stereo
images) and the color texture image of the scene/object without
the superimposed features [See Gu 2 [0006 and last sentence of
0049]].
Id. at 9–10. For independent claims 8 and 20, the Examiner similarly finds
Gu1 and Gu2 disclose the recited elements and articulates similar reasoning
for combining the references. Id. at 11, 13.
Appellants argue, inter alia, the Examiner’s reason for combining the
references does not make sense because Gu2 is directed to improving
performance in stereo imaging by eliminating the need to turn on and off the
projection of a superimposed image onto the scene to be captured but Gu1
does not disclose reliance on any superimposed image for its stereo imaging
depth estimations. App. Br. 13–14. Appellants thus contend that “enabling
the system of Gu1 to take images with superimposed features and images
without superimposed features simultaneously is irrelevant in Gu1.” Id.
We agree with Appellants that the Examiner’s reason for combining
Gu1 and Gu2, to meet these limitations, lacks the requisite explicit findings
and analyses to support a conclusion of obviousness. The Examiner must
support a conclusion of obviousness with explicit findings and analysis
establishing an apparent reason to combine the known elements in the
manner required in the claim at issue. KSR Int’l Co. v. Teleflex Inc., 550
Appeal 2018-008563
Application 13/924,464

8

U.S. 398, 418 (2007). Here, the Examiner identifies no teaching in Gu1 to
make use of active illumination to enhance stereo imaging—i.e., a texture or
pattern superimposed over the scene to be imaged to aid in the accuracy of
trinocular depth estimations. The Examiner fails to explain how reducing
the time delays in switching on and off the projection of such a
superimposed pattern, as taught in Gu2, would benefit the systems and
methods of Gu1.
Accordingly, we are persuaded by Appellants’ argument that the
Examiner has failed to adequately articulate and support a reason that the
ordinarily skilled artisan would have had any reason to consider Gu2’s
disclosure that reduces the time consumption required in turning such active
illumination on and off. In the absence of a sufficiently supported reason to
combine Gu1 and Gu2, we do not sustain the Examiner’s rejection of
independent claims 1, 8, and 20. For the same reasons, we do not sustain the
Examiner’s rejections of dependent claims 4–7, 9–19, 21, and 22.

DECISION
For the above reasons, the Examiner’s decision rejecting all pending
claims 1 and 4–22 is reversed.
REVERSED