Ex Parte EngedalDownload PDFPatent Trial and Appeal BoardDec 7, 201713900158 (P.T.A.B. Dec. 7, 2017) Copy Citation United States Patent and Trademark Office UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS P.O.Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO. 13/900,158 05/22/2013 Torbjorn Engedal P28478USC1 (119-1099USC1) 1015 61947 7590 12/12/2017 Ar>r>le - Rlank Rome. EXAMINER c/o Blank Rome LLP SLATER, ALISON T 717 Texas Avenue, Suite 1400 HOUSTON, TX 77002 ART UNIT PAPER NUMBER 2487 NOTIFICATION DATE DELIVERY MODE 12/12/2017 ELECTRONIC Please find below and/or attached an Office communication concerning this application or proceeding. The time period for reply, if any, is set in the attached communication. Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the following e-mail address(es): mbrininger @ blankrome. com hou stonpatents @ blankrome .com PTOL-90A (Rev. 04/07) UNITED STATES PATENT AND TRADEMARK OFFICE BEFORE THE PATENT TRIAL AND APPEAL BOARD Ex parte TORBJORN ENGEDAL Appeal 2017-008323 Application 13/900,1581 Technology Center 2400 Before CARLA M. KRIVAK, HUNG H. BUI, and JON M. JURGOVAN, Administrative Patent Judges. BUI, Administrative Patent Judge. DECISION ON APPEAL Appellant seeks our review under 35 U.S.C. § 134(a) from the Examiner’s Final Rejection of claims 1—7 and 9—21, which are all the claims pending in the application. We have jurisdiction under 35 U.S.C. § 6(b). We REVERSE.2 1 According to Appellant, the real party in interest is Apple Inc. App. Br. 3. 2 Our Decision refers to Appellant’s Appeal Brief (“App. Br.”) filed January 30, 2017; Reply Brief (“Reply Br.”) filed May 15, 2017; Examiner’s Answer (“Ans.”) mailed March 13, 2017; Final Office Action (“Final Act.”) mailed June 1, 2016; Advisory Action (“Advisory Act.”) mailed December 14, 2016; and original Specification (“Spec.”) filed May 22, 2013. Appeal 2017-008323 Application 13/900,158 STATEMENT OF THE CASE Appellant’s Invention Appellant’s invention relates to a camera pose estimation apparatus and method for augmented reality imaging, for finding a physical camera pose for a camera associated with a head mounted display (HMD) worn by a user. Spec. H 2—3; Title. Appellant’s camera pose estimation enables insertion of virtual objects into an image-stream in real-time by correctly aligning the virtual objects on the screen of the HMD, to obtain an augmented reality effect for the user. Spec. 13; Abstract. Appellant’s apparatus and method determine an estimate for a 3D camera pose relative to a scene from 2D image data of a 2D image frame provided by the camera, using four types of observations: 1. Detected 2D-3D point correspondences (minimize the 2D pixel re-projection error) 2. Tracked 2D-3D point correspondences (minimize the 2D pixel re-projection error) 3. Motion model observations (minimize the translation and rotation relative to previous frame, if the pose was estimated in previous frame) 4. New: Edge observations (minimize the distance between the edges in the captured images and the lines/edges in the 3D model projected to the screen using the current camera pose)[.] By adding the fourth type of observation, the system would be capable of functioning also in environments with little or no texture, but with edges/lines present. Lines/edges will typically be the edge between walls, wall and floor, wall and ceiling, doors and walls etc. Spec. H90-91; see also Fig. 5, Abstract. As such, Appellant’s invention is able to provide robust, real-time 3D camera pose estimation in environments 2 Appeal 2017-008323 Application 13/900,158 lacking texture and contrast, such as “[i]n a room with e.g.[,] only white walls with no paintings or other objects/illustrations.” See Spec, 87—89. Claims 1, 14, and 20 are independent. Representative claim 1 is reproduced below with disputed limitations in italics'. 1. An apparatus for providing an estimate for a 3D camera pose relative to a scene from 2D image data of a 2D image frame provided by a camera, the apparatus comprising: one or more processors; and a memory coupled to the one or more processors and comprising computer readable instructions which, when executed by the one or more processors, cause the one or more processors to: obtain, from a detection thread, detected 2D-3D point correspondences between a current image and a previous image of a scene, obtain, from a tracking thread, tracked 2D-3D point correspondences between the current image and the previous image, obtain motion model observations by comparing the current image to the previous image, obtain edge observations in the current image and the previous image, and determine, in a primary thread, an estimate for a 3D camera pose relative to the scene according to the detected 2D-3D correspondences, the tracked 2D-3D point correspondences, the motion model observations, and the edge observations by minimizing a distance between the edges in the current image and the previous image and edges in a 3D model of the environment projected to a screen based on a previous camera pose. App. Br. 10-13 (Claims App’x). 3 Appeal 2017-008323 Application 13/900,158 Examiner’s Rejections3 & References (1) Claims 1—7, 10, 11, and 13—21 stand rejected under 35 U.S.C. § 103(a) as being unpatentable over L. Vacchetti, V. Lepetit, and P. Fua {Stable Real-Time 3D Tracking Using Online and Offline Information, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26 (10), 1385—1391 (2004); “Fua”) and Comaniciu et al. (US 2003/0174773 Al; Sept. 18, 2003; “Comaniciu”). Final Act. 13-19. (2) Claim 9 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Fua, Comaniciu, and Satoh et al. (US 2006/0094955 Al; published May 4, 2006; “Satoh”). Final Act. 20. (3) Claim 12 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Fua, Comaniciu, and Styers et al. (US 2006/0187034 Al; published Aug. 24, 2006; “Styers”). Final Act. 20-21. 3 Claims 1, 6, 7, 10, and 11 were rejected under 35 U.S.C. § 112, second paragraph, as indefinite. Final Act. 4—5. However, this rejection was withdrawn in the Examiner’s Advisory Action (mailed December 14, 2016), and is no longer pending on appeal. Advisory Act. 2; see also Ans. 2. Additionally, claims 14—19 were rejected under 35 U.S.C. § 101 as directed to non-statutory subject matter. Final Act. 5—6. This rejection, too, was withdrawn in the Examiner’s Advisory Action, and is no longer pending on appeal. Advisory Act. 2; see also Ans. 2. Finally, claims 1—7 and 9—21 were rejected under 35 U.S.C. § 102(b) as anticipated by Engedal (US 2010/0232727 Al; published Sept. 16, 2010). Final Act. 6—12. This rejection was similarly withdrawn in the Examiner’s Advisory Action, and is no longer pending on appeal. Advisory Act. 2; see also Ans. 2. 4 Appeal 2017-008323 Application 13/900,158 Issue on Appeal Based on Appellant’s arguments, the dispositive issue on appeal is whether the combination of Fua and Comaniciu teaches or suggests: obtain, from a detection thread, detected 2D-3D point correspondences between a current image and a previous image of a scene, obtain, from a tracking thread, tracked 2D-3D point correspondences between the current image and the previous image,. . . obtain edge observations in the current image and the previous image, and determine, in a primary thread, an estimate for a 3D camera pose relative to the scene according to the detected 2D- 3D correspondences, the tracked 2D-3D point correspondences, the motion model observations, and the edge observations by minimizing a distance between the edges in the current image and the previous image and edges in a 3D model of the environment projected to a screen based on a previous camera pose, as recited in independent claim 1, and similar limitations recited in independent claims 14 and 20. App. Br. 6—7; Reply Br. 3—5. ANALYSIS Independent claim 1 recites, inter alia: obtain edge observations in the current image and the previous image, and determine, in a primary thread, an estimate for a 3D camera pose relative to the scene according to the detected 2D- 3D correspondences, the tracked 2D-3D point correspondences, the motion model observations, and the edge observations by minimizing a distance between the edges in the current image and the previous image and edges in a 3D model of the environment projected to a screen based on a previous camera pose, 5 Appeal 2017-008323 Application 13/900,158 and further recites “a detection thread” and “a tracking thread” for obtaining the 2D-3D point correspondences. App. Br. 6—7. In support of the obviousness rejection of claim 1, the Examiner finds Fua estimates a 3D camera pose by minimizing a reprojection error for 3D points that are determined from “detected 2D-3D correspondences” and “tracked 2D-3D correspondences” between “incoming images and [target object] keyframes” created in a training stage. Ans. 4 (citing Fua 1386, §§ 3, 3.1, 3.2). The Examiner further finds Fua’s “Related Work” section discloses object tracking that avoids drift by finding camera poses that correctly reproject some fixed features—such as “edges, line segments, or points”—thereby teaching the use of “edge observations“ from current and previous images, as claimed. Ans. 4 (citing Fua 1385, § 2) (emphasis added); Final Act. 14. Based on these findings, the Examiner concludes it would have been obvious to one of ordinary skill to estimate a 3D camera pose in Fua based on a combination of “detected 2D-3D correspondences, tracked 2D-3D correspondences,. . . and edge observations,” as recited in claim 1. Ans. 3^4. Appellant argues “Fua is silent regarding determining a 3D camera pose based on the combination of ‘detected 2D-3D correspondences, tracked 2D-3D correspondences, motion model observations, and edge observations’,” as required by claim 1. App. Br. 7. We agree with Appellant that the Examiner has not identified sufficient evidence to show Fua teaches or suggests the “determining” limitation of claim 1. App. Br. 6— 7; Reply Br. 4—5. In particular, Fua’s “Related Work” section (section 2) merely describes “compar[ing] reprojections from multiple camera poses until the reprojection is suitable” by “fmd[ing] a camera pose that correctly 6 Appeal 2017-008323 Application 13/900,158 reprojects fixed features of a given 3D model, such as edges.” Reply Br. 4— 5 (citing Fua 1385, § 2). In contrast, Appellant’s “claims are directed to identifying a 3D camera pose based on various observations.” Reply Br. 5. In addition, the Examiner has not provided an adequate reason with rational underpinnings to explain why one having ordinary skill in the art would have been led to modify Fua’s tracking rigid objects in 3D with 2D- 3D correspondences for “matched points between keyframe and incoming image” (see Fua 1388, § 5, 1386, § 3, Abstract), to combine edge observations with the 2D-3D correspondences for determining a 3D camera pose as required by claim 1. Fua uses 2D-3D correspondences and back projecting onto a model “to track objects” and determine “the pose of the object. . . based on models of the object,” in contrast to claim 1—directed to “determining an estimate for a f3 jD camera pose relative to a scene.” App. Br. 6 (citing Fua 1386, §§ 3, 3.1, Abstract). See In re Chaganti, 2014, WL 274514, *4 (Fed. Cir. 2014) (“It is not enough to say that... to do so would ‘have been obvious to one of ordinary skill.’. . . Such circular reasoning is not sufficient—more is needed to sustain an obviousness rejection.”) The Examiner additionally finds Fua’s real-time 3D tracking, together with Comaniciu’s “multiple threads that are synchronized” suggest “a primary thread”—for determining an estimate for a 3D camera pose—is separate from “a detection thread” and “a tracking thread” for obtaining detected and tracked 2D-3D point correspondences, as recited in claim 1. Final Act. 14—15 (citing Comaniciu 48, 53). We do not agree. We agree with Appellant that the combination of Fua and Comaniciu does not teach or suggest “a primary thread in which ‘an estimate for a 3D camera pose relative to the scene’ is determined,” and separate specific 7 Appeal 2017-008323 Application 13/900,158 threads by which 2D-3D point correspondences are obtained. App. Br. 7; see also Reply Br. 5. Rather, Comaniciu at most “indicates a modular system where multiple threads may be synchronized.” App. Br. 7. Because the Examiner has not shown the combination of Fua and Comaniciu teaches or suggests all the limitations of claim 1, we do not sustain the Examiner’s obviousness rejection of independent claim 1, and independent claims 14 and 20 reciting similar limitations. See claims 14 and 20 (using “a detection thread” and “a tracking thread,” to “determine, in a primary thread, an estimate for a 3D camera pose for the camera relative to the scene according to the detected 2D-3D correspondences, the tracked 2D- 3D point correspondences, the motion model observations, and the edge observations”). App. Br. 11—13 (Claims App’x). We also do not sustain the Examiner’s rejection of dependent claims 2—7, 9-13, 15—19, and 21 because the Examiner has not shown that the additional teachings of Satoh and Styers make up for the above-noted deficiencies of Fua and Comaniciu. We do not address Appellant’s remaining arguments because the issues discussed supra are dispositive as to all pending claims on appeal. CONCLUSION On the record before us, we conclude Appellant has demonstrated the Examiner erred in rejecting claims 1—7 and 9—21 under 35 U.S.C. § 103(a). DECISION As such, we REVERSE the Examiner’s final rejection of claims 1—7 and 9-21. REVERSED 8 Copy with citationCopy as parenthetical citation