Siemens Medical Solutions USA, Inc.

Patent Trials and Appeals Board

Aug 25, 2021

2020003588 (P.T.A.B. Aug. 25, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/413,486 01/24/2017 Yoshihisa Shinagawa 2015P25049US01 2386
28524 7590 08/25/2021
SIEMENS CORPORATION
IP Dept - Mail Code INT-244
3850 Quadrangle Blvd
Orlando, FL 32817
EXAMINER
KY, KEVIN
ART UNIT PAPER NUMBER
2669
NOTIFICATION DATE DELIVERY MODE
08/25/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
ipdadmin.us@siemens.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________________

Ex parte YOSHIHISA SHINAGAWA
____________________

Appeal 2020-003588
Application 15/413,4861
Technology Center 2600
____________________

Before MARC S. HOFF, ELENI MANTIS MERCADER, and JAMES R.
HUGHES, Administrative Patent Judges.

HOFF, Administrative Patent Judge.

DECISION ON APPEAL

STATEMENT OF THE CASE
Appellant appeals under 35 U.S.C. § 134 from a Final Rejection of
claims 1–20. We have jurisdiction under 35 U.S.C. § 6(b).
We affirm.
Appellant’s invention is a framework for overlaying findings on
medical image data. The framework extracts one or more findings from a
radiology report, and detects one or more anatomical landmarks in image
data corresponding to the radiology report. The one or more extracted

1 Appellants state that the real party in interest is Siemens Medical Solutions
USA, Inc. Appeal Br. 4.
Appeal 2020-003588
Application 15/413,486

2
findings are then correlated to, and overlaid with, the one or more detected
anatomical landmarks on the image data. Spec. ¶ 7.
Claim 1 is reproduced below:
1. One or more non-transitory computer readable media
embodying a program of instructions executable by machine to
perform operations for processing image data, the operations
comprising:
extracting one or more findings from a radiology report
by performing a Natural Language Processing technique,
wherein the one or more extracted findings are represented as
one or more tuples;
assigning, using a processor performing a machine
learning technique, one or more weights to the one or more
extracted findings in accordance with importance or severity;
detecting one or more anatomical landmarks in the image
data corresponding to the radiology report, wherein the one or
more anatomical landmarks are one or more anatomically
meaningful points;
correlating the one or more tuples to the one or more
anatomical landmarks; and
overlaying only the one or more extracted findings
assigned with the one or more weights that are above a
predetermined threshold with the correlated one or more
anatomical landmarks on the image data, wherein the one or
more extracted findings are positioned at the correlated one or
more anatomical landmarks on the image data.

The prior art relied upon by the Examiner as evidence is:
Name Reference Date
Qian US 2014/0149407 A1 May 29, 2014
Ishii US 2014/0323858 A1 Oct. 30, 2014
Parthasarathy US 2015/0324523 A1 Nov. 12, 2015
Liu US 2016/0328643 A1 Nov. 10, 2016

Claims 1, 2, 4–8, 11–13, 17, 19, and 20 stand rejected under
35 U.S.C. § 103 as being unpatentable over Parthasarathy and Ishii.
Appeal 2020-003588
Application 15/413,486

3
Claims 3, 9, 10, and 18 stand rejected under 35 U.S.C. § 103 as being
unpatentable over Parthasarathy, Ishii, and Qian.
Claims 14–16 stand rejected under 35 U.S.C. § 103 as being
unpatentable over Parthasarathy, Ishii, and Liu.
Throughout this decision, we make reference to the Appeal Brief
(“Appeal Br.,” filed Oct. 15, 2016) and the Examiner’s Answer (“Ans.,”
mailed Dec. 12, 2019) for their respective details.
ISSUES
1. Does the combination of Parthasarathy and Ishii teach or suggest
extracting one or more findings from a radiology report, wherein the one or
more extracted findings are represented as one or more tuples?
2. Does the combination of Parthasarathy and Ishii teach or suggest
assigning one or more weights to the one or more extracted findings in
accordance with importance or severity?
3. Does the combination of Parthasarathy and Ishii teach or suggest
detecting one or more landmarks in the image data corresponding to the
radiology report?
4. Does the combination of Parthasarathy and Ishii teach or suggest
overlaying extracted findings assigned with one or more weights that are
above a predetermined threshold with the correlated one or more anatomical
landmarks on the image data?
ANALYSIS
Claims 1, 2, 4–8, 11–13, 17, 19, and 20
Independent claim 1 recites extracting one or more findings from a
radiology report, wherein the one or more extracted findings are represented
as one or more tuples. See Appeal Br. 21 (Claim App.). Independent claims
Appeal 2020-003588
Application 15/413,486

4
1, 5, and 19 each recite assigning one or more weights to the one or more
extracted findings in accordance with importance or severity. See Appeal Br.
21–23, 26–27 (Claim App.). Independent claims 1, 5, and 19 recite detecting
one or more anatomical landmarks in image data, the anatomical landmarks
being anatomically meaningful points. Id. Independent claims 1, 5, and 19
each recite overlaying the extracted findings that are assigned weights above
a predetermined threshold with the correlated anatomical landmarks. Id.
Representing findings as tuples
Appellant first argues that Parthasarathy describes detecting
radiological findings but “does not teach or suggest representing such
findings as tuples.” Appeal Br. 13. We are not persuaded by Appellant’s
argument. Appellant’s Specification defines a tuple as “an ordered list of
elements including, but not limited to, anatomical, disease and/or
pathological terms, possibly together with modifiers.” Spec. ¶ 31. We agree
with the Examiner’s finding that Parthasarathy teaches that “medical
ontologies containing comprehensive medical terms . . . are utilized. The
ontologies are integrated into a NLP [(Natural Language Processing)] to
extract medical terms from narrative reports. . . . [T]he grading device
automatically checks whether there are occurrences of anatomical terms and
measurements (size, shape and extent) in the surrounding text.”
Parthasarathy ¶ 31. We further agree with the Examiner’s finding that
Parthasarathy’s teaching of such medical findings in text form corresponds
to the claimed tuples. Ans. 17.
Weights in accordance with importance or severity
Appellant next contends that Parthasarathy evaluates the severity of
findings, but does not assign one or more weights to the extracted findings
Appeal 2020-003588
Application 15/413,486

5
based on such severity. Appeal Br. 14. Appellant asserts that Parthasarathy
is only concerned with determining how much a medical report contributes
to the current clinical problem. Id., citing Parthasarathy ¶ 32. We are not
persuaded by Appellant’s arguments. As noted supra, we find that
Parthasarathy uses natural language processing to extract medical terms
from narrative reports. Parthasarathy ¶ 31. Parthasarathy further teaches
“evaluat[ing] the severity of the findings. Findings like cancer, carcinoma,
invasive, and the like are reported to referrals. If those findings are present in
a medical report, the system can check whether follow up action are [sic]
also present.” Id. We therefore agree with the Examiner that Parthasarathy
teaches a system in which certain findings are assigned greater
“importance,” which we equate to the claimed “weights.” Ans. 17-18.
Detecting anatomical landmarks
Appellant argues that Parthasarathy fails to teach detecting one or
more anatomical landmarks that are anatomically meaningful points. Appeal
Br. 14–15. Appellant’s contention is not persuasive of error because the
teachings of Parthasarathy, taken together, correspond to this claim
limitation. As noted by Appellant, imaging system 16 generates image data
of a patient obtained using an image diagnosis apparatus. Appeal Br. 14.
System 16 includes display devices 42 and a user interface 44 to adjust
image reconstruction and acquisition parameters and/or for displaying
generated image data to clinicians. Parthasarathy ¶ 20; Ans. 18.
Parthasarathy further teaches a medical report containing “precise
anatomical location, size, extent, shape of the findings.” Parasarathy ¶ 31.
We further observe that Ishii teaches a medical imaging system in
which relevant findings are visually correlated with anatomical locations.
Appeal 2020-003588
Application 15/413,486

6
Ishii Figs. 3, 4. We agree with the Examiner’s finding that, taken together,
these sections of Parthasarathy teach detecting one or more anatomically
meaningful points of a patient. Ans. 18. We also determine that the
combined teachings of Parthasarathy and Ishii suggest detecting one or more
anatomically meaningful points of a patient.
Overlaying sufficiently important findings with anatomical landmarks
Appellant argues that the Examiner erred in relying on Ishii for a
teaching of “overlaying only the extracted findings assigned with the one or
more weights that are above a predetermined threshold with the correlated
one or more anatomical landmarks on the image data.” Appeal Br. 15–16.
We do not agree with Appellant, and we agree with the Examiner’s finding
that Ishii teaches superimposing and displaying an annotation generated by
annotation generation unit 22 on an anatomical image to indicate the target
plaque position on the image. Ans. 19; Ishii ¶ 27, Figs. 3, 4. We further
agree with the Examiner that combining the teachings of Parthasarathy, in
which certain findings “like cancer, carcinoma, invasive, and the like are
reported to referrals” because of their noted severity, with the teachings of
Ishii concerning annotating images with extracted findings, would have
resulted in the claimed invention. Ans. 19.
Hindsight reconstruction
Last, Appellant argues that the Examiner has not provided sufficient
reason why the skilled artisan would combine Parthasarathy and Ishii.
Appeal Br. 17. Appellant alleges that there is no evidence or suggestion in
either reference to produce the claimed configuration. Appeal Br. 18. We do
not agree with Appellant. We agree with the Examiner’s finding that
Parthasarathy and Ishii reside in similar endeavors of medical imaging. Ans.
Appeal 2020-003588
Application 15/413,486

7
20. We determine that the Examiner has stated a rationale, to “improve
medical image diagnostic (sic) by annotating the image including
superimposing a stress value color map on a plaque map,” having a rational
underpinning to support the legal conclusion of obviousness. Final Act. 4;
Ans. 20.
Additionally, although a Reply Brief is not required, Appellant did not
file a Reply Brief and failed to address the Examiner’s clarified findings and
additional discussion of Parthasarathy and Ishii and the disputed limitations
of claim 1, or otherwise rebut the findings and responsive arguments made
by the Examiner in the Answer. See Ans. 17–20.
Accordingly, based on a preponderance of the evidence, we find that
the combination of Parthasarathy and Ishii teaches or suggests all the
limitations of the invention recited in independent claims 1, 5, and 19. Thus,
we sustain the Examiner’s § 103 rejection of claims 1, 2, 4–8, 11–13, 17, 19,
and 20.
Claims 3, 9, 10, and 18
Appellant does not separately argue for the patentability of claims 3,
9, 10, and 18, relying instead on the arguments made for independent claims
1 and 5. We conclude supra that the Examiner did not err in rejecting those
independent claims. We therefore sustain the Examiner’s § 103 rejection of
claims 3, 9, 10, and 18 over Parthasarathy, Ishii, and Qian, for the reasons
given supra with respect to the § 103 rejection of claims 1 and 5.
Claims 14–16
Appellant does not separately argue for the patentability of claims 14–
16, relying instead on the arguments made for independent claim 5. We
conclude supra that the Examiner did not err in rejecting independent claim
Appeal 2020-003588
Application 15/413,486

8
5. We therefore sustain the Examiner’s § 103 rejection of claims 14–16 over
Parthasarathy, Ishii, and Liu, for the reasons given supra with respect to the
§ 103 rejection of claim 5.

CONCLUSIONS
1. The combination of Parthasarathy and Ishii teaches extracting one
or more findings from a radiology report, wherein the one or more extracted
findings are represented as one or more tuples.
2. The combination of Parthasarathy and Ishii teaches assigning one
or more weights to the one or more extracted findings in accordance with
importance or severity.
3. The combination of Parthasarathy and Ishii teaches detecting one or
more landmarks in the image data corresponding to the radiology report.
4. The combination of Parthasarathy and Ishii teaches overlaying
extracted findings assigned with one or more weights that are above a
predetermined threshold with the correlated one or more anatomical
landmarks on the image data.

Appeal 2020-003588
Application 15/413,486

9
DECISION SUMMARY
In summary:
Claims
Rejected
35 U.S.C.
§
Reference(s)/
Basis
Affirmed Reversed
1, 2, 4–8, 11–
13, 17, 19, 20
103 Parthasarathy,
Ishii
1, 2, 4–8,
11–13, 17,
19, 20

3, 9, 10, 18 103 Parthasarathy,
Ishii, Qian
3, 9, 10, 18
14–16 103 Parthasarathy,
Ishii, Liu
14–16
Overall
Result
1–20

ORDER
The Examiner’s decision to reject claims 1–20 is affirmed.
No time period for taking any subsequent action in connection with
this appeal may be extended under 37 C.F.R. § 1.136(a). See 37 C.F.R.
§ 1.136(a)(1)(iv).

AFFIRMED