VMware, Inc.Download PDFPatent Trials and Appeals BoardOct 26, 20212020005576 (P.T.A.B. Oct. 26, 2021) Copy Citation UNITED STATES PATENT AND TRADEMARK OFFICE UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS P.O. Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO. 15/191,415 06/23/2016 Seongbeom KIM C879 5800 152569 7590 10/26/2021 Patterson + Sheridan, LLP - VMware 24 Greenway Plaza Suite 1600 Houston, TX 77046 EXAMINER LOONAN, ERIC T ART UNIT PAPER NUMBER 2137 NOTIFICATION DATE DELIVERY MODE 10/26/2021 ELECTRONIC Please find below and/or attached an Office communication concerning this application or proceeding. The time period for reply, if any, is set in the attached communication. Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the following e-mail address(es): ipadmin@vmware.com psdocketing@pattersonsheridan.com vmware_admin@pattersonsheridan.com PTOL-90A (Rev. 04/07) UNITED STATES PATENT AND TRADEMARK OFFICE ____________ BEFORE THE PATENT TRIAL AND APPEAL BOARD ____________ Ex parte SEONGBEOM KIM, JAGADISH KOTRA, and FEI GUO ____________ Appeal 2020-005576 Application 15/191,4151 Technology Center 2100 _____________ Before HUNG H. BUI, JEREMY CURCURI, and PHILLIP A. BENNETT, Administrative Patent Judges. BUI, Administrative Patent Judge. DECISION ON APPEAL Appellant seeks our review under 35 U.S.C. § 134(a) from the Examiner’s final rejection of claims 1, 3–11, and 13–22, all of the pending claims. Appeal Br. 13–18 (Claims App.). Claims 2 and 12 have been cancelled. We have jurisdiction under 35 U.S.C. § 6(b). We reverse.2 1 “Appellant” herein refers to “applicant” as defined in 37 C.F.R. § 1.42. According to Appellant, VMware Inc. is identified as the real party in interest. Appeal Br. 3. 2 We refer to the Appellant’s Appeal Brief filed February 7, 2020 (“Appeal Br.”); Reply Brief filed July 24, 2020 (“Reply Br.”); Examiner’s Answer mailed May 26, 2020 (“Ans.”); Final Office Action mailed June 6, 2019 (“Final Act.”); and Specification filed June 23, 2016 (“Spec.”). Appeal 2020-005576 Application 15/191,415 2 STATEMENT OF THE CASE Prior art techniques for evaluating resource contention in a non- uniform memory access (NUMA) computer system having “multiple memory proximity domains, referred to as ‘NUMA nodes,’” utilize performance counters to calculate instructions per cycle (IPC) or last level cache (LLC) hit/miss rates. However, using such counters would be inconvenient “for the purpose of NUMA management.” Spec. ¶¶ 1, 3. Appellant’s claimed invention seeks to provide “a more practical technique to evaluate contention at NUMA nodes for process and memory placements, so that better process and memory scheduling decisions can be made in NUMA system.” Spec. ¶ 4. Figure 1, depicting computer system 100 having multiple NUMA nodes 121–122, is reproduced below: Appeal 2020-005576 Application 15/191,415 3 Figure 1 shows computer system 100 having 1st NUMA node 121 and 2nd NUMA node 122, each of which comprises a multi-core processor (CPU cores), memory and memory controller (MC). Spec. ¶ 14. In case of remote memory access, CPU cores of 1st NUMA node 121 access memory of 2nd NUMA node 122 through memory controllers of both NUMA nodes 121, 122, via interconnect 123. System software 110 is provided with latency monitor 111, memory allocator 112, memory rebalancer 113 and rebalancer 114. Spec. ¶ 15. Latency monitor 111 is periodically executed to perform the probing of latencies of local and remote memory accesses at each NUMA node 121 or 122. Spec. ¶ 15. According to Appellant, “latencies of local and remote memory access are probed at each NUMA node . . . If the contention at the local NUMA node increases, the local memory access latency may become large enough so that it becomes beneficial to allocate/rebalance memory to a remote NUMA node. The benefits of using the latency probing technique are its simplicity and accuracy. It does not consume [prior art] performance counters and reflects inter- node distance inherently. It is also capable of capturing contentions on the NUMA interconnect, which cannot be done by using [prior art] performance counter metrics like IPC or LLC hits/misses.” Spec. ¶ 4. For example, “if there is an imbalance in contention of resources between two NUMA nodes, the NUMA [] [allocator 112] executes either [1] memory rebalancer 113 to migrate allocated memory pages between two NUMA nodes or [2] computer rebalancer 114 to migrate processes and their memory pages between the two NUMA nodes.” Spec. ¶ 15. In particular, latency monitor 111 (1) reads memory pages from the local NUMA node 121 (step 206, Figure 2), (2) reads memory pages from each of the remote Appeal 2020-005576 Application 15/191,415 4 NUMA nodes 122 (step 208, Figure 2), and (3) measures the latency of each memory access for storage in either the local NUMA node 121 or the remote NUMA node (step 210, Figure 2). Spec. ¶ 18, Figure 2. Representative Claim Claims 1, 11, and 19 are independent claims. Representative claim 1 is reproduced below with disputed limitations emphasized for clarity: 1. In a computer system having multiple memory proximity domains including a first memory proximity domain with a first processor, a first memory controller, and a first memory and a second memory proximity domain with a second processor, a second memory controller, and a second memory, a method of managing placement of memory pages associated with a process in one of the memory proximity domains, said method comprising: reading memory pages by the first processor from the first memory using the first memory controller and not the second memory controller; reading memory pages by the first processor from the second memory using the first memory controller and the second memory controller; measuring, by a latency monitor, (1) a first one or more latencies of memory reads during the reading of memory pages by the first processor from the first memory using the first memory controller and not the second memory controller, and (2) a second one or more latencies of memory reads during the reading of memory pages by the first processor from the second memory using the first memory controller and the second memory controller; comparing the measured first one or more latencies to the measured second one or more latencies; and based on the comparing, placing memory pages into the second memory proximity domain, wherein the placed memory pages are associated with a first process running on the first Appeal 2020-005576 Application 15/191,415 5 processor, so that after said placing, the first process is accessing the memory pages from the second memory during execution. Appeal Br. 13 (Claims App.). REJECTIONS AND REFERENCES (1) Claims 1, 3–5, 8–11, 13–15, and 18–20 stand rejected under 35 U.S.C. § 103 as obvious over the combined teachings of Michalak (US 2014/0136773 A1; issued May 15, 2014) and Schoinas et al. (US 6,347,362 B1; published Feb. 12, 2002; “Schoinas”). Final Act. 2–8. (2) Claims 6 and 16 stand rejected under 35 U.S.C. § 103 as obvious over the combined teachings of Michalak, Schoinas, and Brooker et al. (US 8,918,392 B1; issued Dec. 23, 2014; “Brooker”). Final Act. 8–9. (3) Claims 7 and 17 stand rejected under 35 U.S.C. § 103 as obvious over the combined teachings of Michalak, Schoinas, and Vaid et al. (US 2010/0121865 A1; published May 13, 2010). Final Act. 9–10. (4) Claims 21 and 22 stand rejected under 35 U.S.C. § 103 as obvious over the combined teachings of Michalak, Schoinas, and Arimilli et al. (US 2003/0009640 A1; published Jan. 9, 2003; “Arimilli”). Final Act. 10–11. ANALYSIS In support of the obviousness rejection, the Examiner finds the combination of Michalak and Schoinas teaches all limitations of Appellant’s claims 1, 11, and 19. Final Act. 2–4. In particular, the Examiner finds Michalak teaches all steps recited in Appellant’s claimed “computer system” Appeal 2020-005576 Application 15/191,415 6 including: (1) “reading memory pages . . . from the first memory using the first memory controller and not the second memory controller” (Final Act. 2 (citing Michalak ¶ 41, Figure 4)); (2) “reading memory pages . . . from the second memory using the first memory controller and the second memory controller;” (id. 3 (citing Michalak ¶ 41, Figure 4)); (3) “measuring, by a latency monitor, (1) a first one or more latencies of memory read [] from the first memory[] and (2) a second one or more latencies of memory read [] from the second memory[]” (id. 3 (citing Michalak ¶ 46, Figure 4)); (4) “comparing the measured . . . latencies;” and (5) “based on the comparing, placing memory pages into the second memory proximity domain.” (id. at 3 (citing Michalak ¶ 10)). The Examiner acknowledges Michalak’s computer system is not a NUMA system, but relies on Schoinas for teaching a NUMA system having multiple memory proximity domains, referred to as “NUMA nodes” each of which comprises a multi-core processor (CPUs), memory and memory controller (MC) in order to support the conclusion of obviousness, i.e., “it would have been obvious . . . to modify the computer system of Michalak to include a plurality of nodes as taught by Schoinas [] to permit sharing of main memory among a plurality of nodes.” Final Act. 3–4 (citing Schoinas 1:14–27, 2:56–59, 5:6–7, Figure 1). Because Michalak’s computer system is not a NUMA system having multiple NUMA nodes with specific controllers, Appellant argues Michalak cannot and does not teach the disputed limitations of claims 1, 11, and 19, which require interactions of specific controllers of a NUMA system, including, for example: Appeal 2020-005576 Application 15/191,415 7 reading memory pages by the first processor from the second memory using the first memory controller and the second memory controller; [and] measuring, by a latency monitor, (1) a first one or more latencies of memory reads during the reading of memory pages by the first processor from the first memory using the first memory controller and not the second memory controller, and (2) a second one or more latencies of memory reads during the reading of memory pages by the first processor from the second memory using the first memory controller and the second memory controller. Appeal Br. 8–9; Reply Br. 2–3. Likewise, Appellant argues “Schoinas does not teach or suggest measuring latency and does not cure the deficiencies of Michalak.” Appeal Br. 9. (Capital emphasis omitted.) The Examiner does not sufficiently respond to Appellant’s arguments. Instead, the Examiner takes the position that while Michalak does not teach “measuring latencies of memory reads to a second memory using both controllers as claimed,” (Ans. 6) Schoinas teaches access paths of (1) “reading memory pages by the first processor from the first memory using the first memory controller and not the second memory controller” (Path 1 of Schoinas) and (2) “reading memory pages by the first processor from the second memory using the first memory controller and the second memory controller” (Path 2 of Schoinas) (Ans. 7) as proposed in Schoinas’ marked- up Figure 1 below. Schoinas’ Figure 1, depicting the Examiner’s proposed access to local memory and remote memory in NUMA system 10, is reproduced below with additional markings for illustration. Appeal 2020-005576 Application 15/191,415 8 Schoinas’ Figure 1, as reproduced above, depicts NUMA system 10 including a plurality of NUMA nodes each of which includes CPUs 12, memory 14, and node/memory controller 14. Schoinas 1:6–11; 2:56–62. According to the Examiner, Schoinas’ CPU 12, shown in Figure 1, can access paths of (1) “reading memory pages [] from the first memory [16] using the first memory controller [14]” (Path 1) and (2) “reading memory pages [] from the second memory [16] using the first memory controller [14] and the second memory controller [14]” (Path 2). Ans. 7. We agree with the Examiner’s position regarding Schoinas only in part. Obviousness is a question of law based on underlying factual findings, Appeal 2020-005576 Application 15/191,415 9 In re Baxter, 678 F.3d 1357, 1361 (Fed. Cir. 2012), including what a reference teaches, In re Beattie, 974 F.2d 1309, 1311 (Fed. Cir. 1992), and the existence of a reason to combine references, In re Hyon, 679 F.3d 1363, 1365–66 (Fed. Cir. 2012). At the outset, we note both Michalak and Schoinas describe prior art techniques, as acknowledged by Appellant’s own admitted prior art (AAPA), that utilize performance counters to calculate instructions per cycle (IPC) or last level cache (LLC) hit/miss rates to evaluate resource contention in a computer system. Spec. ¶¶ 1, 3. For example, Michalak teaches “hardware-based page access counters [] used to record memory utilization [in a mobile processor].” Michalak ¶¶ 1, 7. According to Michalak, based on the utilizations measured with the page access counters 480, the kernel or operating system routine may change virtual address to physical address mappings used in the MMU 450 to dynamically assign (or reassign) pages to the most efficient memory available in the main memory associated with the processor 440. In one embodiment, relative efficiencies associated with different memory types that may be available in the main memory may be defined based on capacities, throughputs, latencies, and/or power efficiencies associated therewith. In particular, with respect to a given memory, capacity may generally refer to total storage available therein (typically expressed in GB), throughput may refer to the rate to transfer data over the memory interface associated therewith (typically expressed in GB/s or gigabytes per second), latency may refer to a portion of the cache miss penalty that represents a time for the MMU 450 to access the memory during a load instruction from the processor 440, and power efficiency may refer to the power that the memory consumes during idle, active, or other I/O states. Accordingly, in one embodiment, the kernel or operating system routine may be configured with predefined rules to specify relative efficiencies associated with certain existing memory types and therefore define the "more efficient" Appeal 2020-005576 Application 15/191,415 10 and "less efficient" memory types based thereon . . . Alternatively (or additionally), the kernel or operating system routine may run one or more appropriate benchmarking processes to measure the relative capacities, throughputs, latencies, and power efficiencies associated with the different memory types available in main memory and define the memory types having greater or lesser efficiency based thereon. Michalak ¶ 46 (emphasis added). As such, all the cited paragraphs 10, 41, and 46 of Michalak pertain to (1) monitoring page access counters, and (2) dynamically assigning the pages in a first memory and the pages in a second memory based on the monitored page access counters. Michalak ¶¶ 10, 41, and 46. As correctly recognized by Appellant, these cited paragraphs of Michalak do not teach the steps recited in Appellant’s claimed “computer system,” especially “the specific measuring of latency [via latency monitor 111 of system software 110, shown in Appellant’s Figure 1] during the specific reads with the specific controllers as claimed.” Appeal Br. 8. Like Michalak, Schoinas also teaches monitoring counters in the context of a NUMA system having multiple NUMA nodes to track certain types of event. Schoinas 2:13–21, 3:1–5, Figure 2. However, as recognized by Appellant, “Schoinas does not teach or suggest measuring latency and does not cure the deficiencies of Michalak” to arrive at Appellant’s claimed invention. Appeal Br. 9. As a NUMA system provided with multiple NUMA nodes, we agree with the Examiner that Schoinas’ CPU 12, shown in Figure 1, is capable of accessing paths of (1) “reading memory pages [] from the first memory [16] using the first memory controller [14]” (Path 1) and (2) “reading memory pages [] from the second memory [16] using the first memory controller [14] Appeal 2020-005576 Application 15/191,415 11 and the second memory controller [14]” (Path 2). Ans. 7. However, as recognized by Appellant, “nothing in Fig. 1 of Schoinas or its brief description in col. 2, lines 55-67, that teaches or suggests managing placement of memory pages in the NUMA system.” Reply Br. 2. In fact, neither Michalak nor Schoinas, alone or in combination, teaches or suggests the use of Appellant’s claimed “latency monitor” as a software solution for probing and measuring latencies during the specific reads with the specific controllers, i.e., (1) a first one or more latencies of memory reads during the reading of memory pages by the first processor from the first memory using the first memory controller and not the second memory controller, and (2) a second one or more latencies of memory reads during the reading of memory pages by the first processor from the second memory using the first memory controller and the second memory controller as recited in claims 1, 11, and 19. This way, the use of prior art “counters” such as disclosed in Michalak and Schoinas could be avoided as acknowledged by AAPA. Spec. ¶ 3. Lastly, we also note the Examiner’s rationale to incorporate the teachings of Schoinas into Michalak is insufficient to support the combination as required by KSR Int’l Co. v. Teleflex, Inc., 550 U.S. 398, 418 (2007); and In re Kahn, 441 F.3d 977, 988 (Fed Cir. 2006) (“[R]ejections on obviousness grounds cannot be sustained by mere conclusory statements; instead, there must be some articulated reasoning with some rational underpinning to support the legal conclusion of obviousness.”). For example, the Examiner reasons “it would have been obvious . . . to modify the computer system of Michalak to include a plurality of nodes as taught by Schoinas [] to permit sharing of main memory among a plurality of nodes.” Appeal 2020-005576 Application 15/191,415 12 Final Act. 3–4. However, incorporating Schoinas’ NUMA system into a mobile computer system of Michalak alone is insufficient to arrive at Appellant’s claimed “latency monitor” to measure the latency during the specific reads with the specific controllers in the manner recited in claims 1, 11, and 19. For these reasons, we do not sustain the Examiner’s obviousness rejection of independent claims 1, 11, and 19, and the rejections of their respective dependent claims 2, 8–12, 14, 15, and 17–19, which are not argued separately. CONCLUSION On the record before us, we conclude Appellant has demonstrated the Examiner erred in rejecting (1) claims 1, 3–5, 8–11, 13–15, and 18–20 as obvious over the combined teachings of Michalak and Schoinas; (2) claims 6 and 16 as obvious over the combined teachings of Michalak, Schoinas, and Brooker; (3) claims 7 and 17 as obvious over the combined teachings of Michalak, Schoinas, and Vaid; and (4) claims 21 and 22 as obvious over the combined teachings of Michalak, Schoinas, and Arimilli. DECISION SUMMARY In summary: Claims Rejected 35 U.S.C. § References/Basis Affirmed Reversed 1, 3–5, 8– 11, 13–15, 18–20 103 Michalak, Schoinas 1, 3–5, 8–11, 13–15, 18– 20 6, 16 103 Michalak, Schoinas, Brooker 6, 16 Appeal 2020-005576 Application 15/191,415 13 Claims Rejected 35 U.S.C. § References/Basis Affirmed Reversed 7, 17 103 Michalak, Schoinas, Vaid 7, 17 21, 22 103 Michalak, Schoinas, Arimilli 21, 22 Overall Outcome 1, 3–11, 13– 22 REVERSED Copy with citationCopy as parenthetical citation