Microsoft Technology Licensing, LLC

Patent Trials and Appeals Board

Aug 5, 2020

2019005538 (P.T.A.B. Aug. 5, 2020)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/347,653 11/09/2016 Ann M. Paradiso 14917.2996US01/400381USNP 3791
27488 7590 08/05/2020
MERCHANT & GOULD (MICROSOFT)
P.O. BOX 2903
MINNEAPOLIS, MN 55402-0903
EXAMINER
PULLIAS, JESSE SCOTT
ART UNIT PAPER NUMBER
2657
NOTIFICATION DATE DELIVERY MODE
08/05/2020 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
USPTO27488@merchantgould.com
usdocket@microsoft.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________________

Ex parte ANN M. PARADISO, JONATHAN CAMPBELL,
EDWARD BRYAN CUTRELL, HARISH KULKARNI,
MEREDITH MORRIS, ALEXANDER JOHN FIANNACA, and
KILEY REBECCA SOBEL
____________________

Appeal 2019-005538
Application 15/347,653
Technology Center 2600
____________________

Before ROBERT E. NAPPI, LINZY T. McCARTNEY, and
JAMES W. DEJMEK, Administrative Patent Judges.

DEJMEK, Administrative Patent Judge.

DECISION ON APPEAL
Appellant1 appeals under 35 U.S.C. § 134(a) from a Final Rejection of
claims 1–8, 13–15, and 18. The Examiner has objected to claims 9–12, 16,
17, 19, and 20 as being dependent upon a rejected base claim, but has
indicated these claims would be allowable if rewritten in independent form.

1 Throughout this Decision, we use the word “Appellant” to refer to
“applicant” as defined in 37 C.F.R. § 1.42 (2018). Appellant identifies
Microsoft Technology Licensing, LLC as the real party in interest. Appeal
Br. 4.
Appeal 2019-005538
Application 15/347,653
2
See Final Act. 12. We have jurisdiction over the remaining pending claims
under 35 U.S.C. § 6(b).
We affirm.
STATEMENT OF THE CASE
Introduction
Appellant’s disclosed and claimed invention generally relates to
“providing [an] intuitive synthesized speech-authoring user interface for
generating expressive content.” Spec. ¶ 11. More specifically, Appellant
describes an expressive synthesized speech system as comprising an
expressive keyboard in which a user may input text and select an expressive
operator (e.g., an emoji). Spec. ¶¶ 15–17, 50. The selection of the
expressive operator serves to change the tone, for example, of voice
synthesized from the user’s input text to correspond to the selected
expressive operator. Spec. ¶ 15.
Claims 1, 13, and 18 are independent claims. Claim 1 is
representative of the subject matter on appeal (see 37 C.F.R.
§ 41.37(c)(1)(iv)) and is reproduced below:
1. A computer-implemented method for generating
expressive content comprising:
displaying an expressive keyboard, wherein the expressive
keyboard includes an alphanumeric keyboard for receiving
textual input and a plurality of expressive operators for
selectively applying an emotional tone or a vocal sound effect
associated with each of the plurality of expressive operators to
received textual input;
receiving textual input;
displaying on the expressive keyboard a set of the plurality
of expressive operators, based on linguistic properties of the
received textual input;
Appeal 2019-005538
Application 15/347,653
3
selecting, by a user, at least one of the displayed plurality
of expressive operators;
in response to receiving the selection of at least one of an
expressive operator, identifying a predefined set of one or more
prosody attributes or vocal sound effects associated with the
selected expressive operator;
combining the associated predefined set of prosody
attributes or the vocal sound effect with the received textual
input; and
outputting the combined set of prosody attributes or the
vocal sound effect and textual input to a speech generation
engine for generating expressive synthesized speech.
The Examiner’s Rejections
1. Claims 1–4, 6–8, 13–15, and 18 stand rejected under 35 U.S.C.
§ 103 as being unpatentable over Radebaugh (US 2014/0067397 A1; Mar. 6,
2014) and Grieves et al. (US 2015/0100537 A1; Apr. 9, 2015) (“Grieves”).
Final Act. 5–11
2. Claim 5 stands rejected under 35 U.S.C. § 103 as being
unpatentable over Radebaugh, Grieves, and Liu et al. (US 2017/0083506
A1; Mar. 23, 2017) (“Liu”). Final Act. 11–12.
ANALYSIS2
Independent claims 1, 13, and 18
Appellant asserts the independent claims require two sources of
input—textual input and a user-selected expressive operator from an
expressive keyboard. Appeal Br. 10. Appellant argues Radebaugh, as relied
on by the Examiner, only requires textual input, which is converted to output

2 Throughout this Decision, we have considered the Appeal Brief, filed
April 16, 2019 (“Appeal Br.”); the Reply Brief, filed July 8, 2019 (“Reply
Br.”); the Examiner’s Answer, mailed May 24, 2019 (“Ans.”); and the Final
Office Action, mailed November 21, 2018 (“Final Act.”), from which this
Appeal is taken.
Appeal 2019-005538
Application 15/347,653
4
speech. Appeal Br. 10 (citing Radebaugh ¶¶ 5, 40, Figs. 1A, 2). That is,
Appellant argues Radebaugh does not teach using a user-selected expressive
operator from an expressive keyboard to add speech expressivity to the
generated output speech. Appeal Br. 10–14. Instead, Appellant asserts
Radebaugh adds speech expressivity based only on the content of the input
text. Appeal Br. 10–14 (citing Radebaugh ¶¶ 5–8, 25–26, 34, 53, Figs. 1A,
2). Moreover, Appellant asserts Radebaugh fails to teach or suggest the
claimed expressive keyboard. Appeal Br. 11–14; Reply Br. 5–6. Appellant
argues that, at most, Radebaugh merely describes a conventional keyboard.
Appeal Br. 13 (citing Radebaugh ¶ 57).
In addition, Appellant argues Grieves fails to remedy the deficiencies
of Radebaugh. Appeal Br. 14–16; Reply Br. 3–6. In particular, Appellant
asserts Grieves also fails to describe an expressive keyboard. Appeal
Br. 14–16; Reply Br. 3–6. Instead, Appellant argues Grieves discloses a
conventional keyboard “for a user authoring a text message or an email,
including the ability to insert emotive icons, such as a smiley face, into the
text.” Appeal Br. 15 (citing Grieves ¶ 25, Fig. 3). Appellant argues Grieves
is silent regarding the claimed feature of applying any prosody attributes
based on the inserted emotive icons to a generated audio output. Appeal
Br. 14–16; Reply Br. 5–6.
Claim construction is an important step in a patentability
determination. A legal conclusion that a claim is obvious involves two
analytical steps, assuming the references have been properly combined
under § 103. See Medichem, S.A. v. Rolabo, S.L., 353 F.3d 928, 933 (Fed.
Cir. 2003) (“Both anticipation under § 102 and obviousness under § 103 are
two-step inquiries. The first step in both analyses is a proper construction of
the claims. . . . The second step in the analyses requires a comparison of the
Appeal 2019-005538
Application 15/347,653
5
properly construed claim to the prior art.” (internal citations omitted)).
Under the second step, the Board must compare the construed claim to one
or more prior art references and make factual findings regarding the
limitations contested by Appellant. See In re Crish, 393 F.3d 1253, 1256
(Fed. Cir. 2004).
When construing claim terminology during prosecution before the
Office, claims are to be given their broadest reasonable interpretation
consistent with the Specification, reading claim language in light of the
Specification as it would be interpreted by one of ordinary skill in the art.
In re Am. Acad. of Sci. Tech Ctr., 367 F.3d 1359, 1364 (Fed. Cir. 2004). We
are mindful, however, that limitations are not to be read into the claims from
the Specification. In re Van Geuns, 988 F.2d 1181, 1184 (Fed. Cir. 1993).
Because “applicants may amend claims to narrow their scope, a broad
construction during prosecution creates no unfairness to the applicant or
patentee.” In re ICON Health & Fitness, Inc., 496 F.3d 1374, 1379 (Fed.
Cir. 2007) (citation omitted).
Claim 1 describes an expressive keyboard as “includ[ing] an alpha-
numeric keyboard for receiving textual input and a plurality of expressive
operators for selectively applying an emotional tone or a vocal sound effect
associated with each of the plurality of expressive operators to received
textual input.” The Specification describes an expressive operator may be
either an emoji object or a punctuation object. Spec. ¶ 50. In addition, we
note dependent claims 2 and 8 further refine the expressive operators as
comprising (i) a plurality of emoji objects wherein an emoji object is
illustrative of an emotion, and (ii) a plurality of punctuation objects. Thus,
we conclude an expressive keyboard comprises an alpha-numeric keyboard
Appeal 2019-005538
Application 15/347,653
6
and a plurality of emoji and/or punctuation objects that are used to apply a
desired expressive effect to received textual input.
Radebaugh generally relates to synthesizing speech from a text input
and using emoticons identified in the text input to provide contextual text-to-
speech expressivity. Radebaugh ¶ 5, Abstract. Radebaugh describes an
expressivity manager that identifies one or more emoticons within an input
text string and tags the text string with “an expressivity tag that indicates
expressivity corresponding to the emoticon.” Radebaugh ¶ 7. Expressivity
may include “changes in intonation, prosody, speed, pauses, and other
features.” Radebaugh ¶ 6. Radebaugh describes exemplary emoticons such
as a “smiley face” formed by a colon immediately followed by a hyphen and
then immediately followed by a right parenthesis. Radebaugh ¶ 5. As set
forth in Radebaugh, an emoticon “refers to any combination of punctuation
marks and/or characters appearing in a character or text string used to
express a person’s mood.” Radebaugh ¶ 34. Further, Radebaugh
contemplates that an emoticon may also include “graphics or images within
text used to convey tone or mood, such as emoji or other picture characters
or pictograms.” Radebaugh ¶ 34.
Figure 1A of Radebaugh is illustrative and is reproduced below:
Appeal 2019-005538
Application 15/347,653
7

Figure 1A of Radebaugh is a block diagram of the disclosed text-to-
speech system. Radebaugh ¶ 18. As shown, a text input string includes an
emoticon (i.e., a “sad face”). Radebaugh ¶ 25. Text-to-speech engine (105)
identifies the emoticon and adapts the expressivity of the generated audio to
read out the text string in a sad voice. Radebaugh ¶ 25, Fig. 1A.
Grieves generally relates to text prediction that considers emoji as
well as words and phrases. Grieves ¶ 4. Specifically, during interaction
with a device, entry of characters is detected and prediction candidates
corresponding to the detected characters are generated. Grieves ¶ 21.
Figure 3 of Grieves is illustrative and is reproduced below:
Appeal 2019-005538
Application 15/347,653
8

Figure 3 of Grieves illustrates “an example prediction scenario
including emoji.” Grieves ¶ 9. As shown, a user is responding to a text
message inquiring about an estimated time of arrival (i.e., “eta”). See
Grieves, Fig. 3. As illustrated, the user has started entering a response of
“Running late, be.” Grieves, Fig. 3. Figure 3 shows a text prediction bar
(306), which may include words (130) or emoji (132), that the user may
select to complete the response. Grieves ¶ 35. The predicted options “may
be configured as selectable elements of the user interface.” Grieves ¶ 36.
Figure 3 also illustrates the keyboard (302) includes an emoji key (310) that
“may be operable to expose an emoji picker to facilitate browsing and
selection of emoji for a message/document from among a library of available
emoji.” Grieves ¶ 37; see also Grieves, Figs. 7, 8 (illustrating a plurality of
emoji).
Given our review of both Radebaugh and Grieves supra, we do not
find Appellant’s arguments persuasive of Examiner error. Rather, although
Appeal 2019-005538
Application 15/347,653
9
Figure 1A of Radebaugh does not expressly show a keyboard, Radebaugh
describes a computer system including a keyboard. See Radebaugh ¶¶ 57,
59, Fig. 5. At a minimum, we find Radebaugh suggests using a keyboard to
provide the text input to the text-to-speech engine of Figure 1A. See also
Ans. 8. Because Radebaugh describes the text input may also comprise
emoticons, we find the keyboard would also include a plurality of
punctuation objects to input the emoticon (e.g., the sad face “:-(” of
Fig. 1A). In addition, Radebaugh describes that the tone and mood of the
audio output is influenced by the emoticon (i.e., plurality of punctuation
objects). See Radebaugh, Fig. 1A.
Moreover, we find Grieves teaches an alpha-numeric keyboard
including a plurality of punctuation objects and emoji objects. For example,
we note in Figure 3 of Grieves the keyboard is expanded to include a text
prediction (306) bar showing a plurality of emoji objects (a house and a
smiley face) for possible insertion into the text. The keyboard also includes
an emoji key (310) that may expose an emoji picker to select an emoji from
among a library of available emoji. Greives ¶ 37; see also Grieves, Fig. 8.
We also agree with the Examiner that Grieves’ keyboard provides for a
plurality of punctuation objects. See, e.g., Ans. 6 (noting that the text box of
Figure 3 displays a comma (which the user has entered) in addition to a
period on the keyboard).
To the extent Appellant asserts Radebaugh does not teach a plurality
of emoji objects, or Grieves fails to teach applying an emotional tone or
vocal sound effect to received textual input (see, e.g., Appeal Br. 10–16), we
are unpersuaded of Examiner error. Non-obviousness cannot be established
by attacking references individually where, as here, the ground of
unpatentability is based upon the teachings of a combination of references.
Appeal 2019-005538
Application 15/347,653
10
In re Keller, 642 F.2d 413, 426 (CCPA 1981). Rather, the test for
obviousness is whether the combination of references, taken as a whole,
would have suggested the patentee’s invention to a person having ordinary
skill in the art. In re Merck & Co., 800 F.2d 1091, 1097 (Fed. Cir. 1986).
Thus, we agree with the Examiner and determine that the combined
teachings and suggestions of Radebaugh and Grieves teach an expressive
keyboard comprising an alpha-numeric keyboard and a plurality of emoji
and/or punctuation objects that are used to apply a desired expressive effect
to received textual input.
In addition, contrary to Appellant’s assertions (see Appeal Br. 23–25;
Reply Br. 11–12), we find the Examiner has provided the requisite
“articulated reasoning with some rational underpinning” to support the
proposed combination of Radebaugh and Grieves teaching the claimed
expressive keyboard. In re Kahn, 441 F.3d 977, 988 (Fed. Cir. 2006) (cited
with approval in KSR Int’l Co. v. Teleflex, Inc., 550 U.S. 398, 418 (2007)).
In particular, the Examiner determines that an ordinarily skilled artisan
would have been motivated to modify the expressive keyboard suggested by
Radebaugh to further include the emojis provided by the keyboard of
Grieves “to reduce difficulty for users to use emoji” instead of, or in addition
to, the emoticons of Radebaugh. Final Act. 6 (citing Grieves ¶ 3); Ans. 7.
For the reasons discussed supra, we are unpersuaded of Examiner
error. Accordingly, we sustain the Examiner’s rejection of independent
claim 1. For similar reasons, we also sustain the Examiner’s rejection of
independent claims 13 and 18, which recite commensurate limitations and
were not argued separately. See Appeal Br. 8–16; see also 37 C.F.R.
§ 41.37(c)(1)(iv).

Appeal 2019-005538
Application 15/347,653
11
Dependent claims 2–8, 14, and 15
a. Claim 2
Claim 2 recites “wherein displaying the expressive keyboard
including the plurality of expressive operators comprises displaying a
plurality of emoji objects, wherein each emoji object is illustrative of an
emotion.”
Appellant asserts Figure 3 of Grieves shows a single emoji key (310)
and that this key is only operable to facilitate selection of a corresponding
emoji to replace a word or text string in a message. Appeal Br. 17 (citing
Grieves ¶¶ 37–38); Reply Br. 7–8.
We disagree. As discussed above with reference to the independent
claims, we find Grieves describes a keyboard comprising a plurality of emoji
objects. See also Grieves, Figs. 3, 8 (illustrating a smiley face menu to
select from a plurality of emoji objects illustrative of an emotion).
Accordingly, we sustain the Examiner’s rejection of claim 2.
b. Claims 3 and 4
Claim 3 depends from claim 2 and recites “displaying a plurality of
emoji objects for selectively providing a visual effect associated with a
selected emoji object as output.”
Appellant argues “Grieves is completely silent regarding providing
any output visual effect associated with the selected emoji object and
certainly does not disclose any visual emoji or effect on the screen of
Fig. 3.” Appeal Br. 17; Reply Br. 7–8.
In response, the Examiner finds, and we agree, Figure 3 of Grieves
illustrates the inclusion of an emoji object to a text message to provide a
visual effect associated with a selected emoji object as output. See Ans. 5–6.
Appeal 2019-005538
Application 15/347,653
12
For example, as shown in Figure 3 of Grieves, the displayed text message
includes a heart emoji object.
Accordingly, we sustain the Examiner’s rejection of claim 3.
Additionally, we sustain the Examiner’s rejection of claim 4, which depends
therefrom and was not argued separately. See Appeal Br. 17; see also
37 C.F.R. § 41.37(c)(1)(iv).
c. Claims 6 and 7
Claims 6 and 7 each depend from claim 1 and provide additional
detail to providing the expressive aspect to the received textual input. For
example, claim 6 recites “wherein combining the associated predefined set
of prosody attributes with the received textual input comprises applying at
least one of pause length, pitch, speed, and emphasis properties to the textual
input” and claim 7 identifies various vocal sound effects (e.g., a laugh).
Appellant argues that because Radebaugh fails to teach the claimed
expressive keyboard, Radebaugh cannot possibly teach the recited features
of claims 6 and 7. Appeal Br. 18; Reply Br. 8–9.
As discussed above with respect to the independent claims, the
Examiner relies on the combined teachings and suggesting of Radebaugh
and Grieves to teach the claimed expressive keyboard. See, e.g., Ans. 4; see
also Keller, 642 F.2d at 426. Moreover, Radebaugh expressly teaches
modifying the expressivity of the synthesized audio output “such as by
changing intonation, prosody, speed, pauses, and other expressivity
characteristics” as well as the application of vocal effects such as “audible
happiness, laughs, sadness, and so forth”). Radebaugh ¶ 34, Abstract.
Accordingly, we sustain the Examiner’s rejection of claims 6 and 7.

Appeal 2019-005538
Application 15/347,653
13
d. Claim 8
Claim 8 depends from claim 1 and recites “wherein displaying the
expressive keyboard including the plurality of expressive operators
comprises displaying a plurality of punctuation objects.”
Appellant asserts Figure 3 of Grieves only shows a keyboard
comprising a single period—i.e., not a plurality of punctuation objects.
Appeal Br. 18; Reply Br. 9. Additionally, replying to the Examiner’s
finding that text box (304) of Grieves, Figure 3 includes a comma (see
Ans. 6), Appellant argues the comma is “merely a punctuation element of
the text message.” Reply Br. 9.
We do not find Appellant’s arguments persuasive of Examiner error.
As discussed above with reference to the independent claims, we find the
emoticons of Radebaugh to be at least suggestive of a keyboard comprising
a plurality of punctuation objects. Additionally, as identified by the
Examiner, the text message response being drafted by the user of the
keyboard in Figure 3 of Grieves includes a comma. Thus, we disagree that it
is merely a punctuation element of a text message and illustrative of a
Grieves’ keyboard providing a means to enter the punctuation object (i.e.,
the comma) into the text message.
Accordingly, we sustain the Examiner’s rejection of claim 8.
e. Claims 14 and 15
Appellant acknowledges that claim 14 “recites features that comprise
a combination of the features recited in claims 2, 3, and 4” and that claim 15
“recites a feature that corresponds to the subject matter recited in claim 6.”
Appeal Br. 18–19. As such, Appellant relies on similar arguments as those
presented with respect to claims 2–4 and 6.
Appeal 2019-005538
Application 15/347,653
14
For the reasons discussed above, we do not find Appellant’s
arguments persuasive of Examiner error. Accordingly, we sustain the
Examiner’s rejection of claims 14 and 15.
f. Claim 5
Claim 5 depends from claim 2 and recites, in relevant part
“determining a set of emoji objects to display based on data associated with
a user’s emotional state, wherein the set includes emoji objects associated
with an emotion corresponding to the user’s emotional state; and displaying
the set of emoji objects.” The Examiner relies on Liu to teach these
limitations. Final Act. 11 (citing Liu, Abstract).
Appellant admits “the system of Liu does determine at least one emoji
character for suggesting to the user based on the user’s emotional state, and
the suggested emoji character is selected by the user to be added or
appended to the user’s entered text,” but asserts the selected emoji fails to
provide for the emotional tone or vocal sound effect recited in independent
claim 1. Appeal Br. 21; Reply Br. 9.
As discussed above with respect to the independent claims, the
Examiner relies on the combined teachings of Radebaugh and Grieves to
teach the claimed expressive keyboard and application of modified prosody
attributes or vocal sound effects to received textual input to generate
expressive synthesized speech. Thus, Appellant’s arguments that Liu fails to
provide for the emotional tone or vocal sound effects is not responsive to the
rejection as articulated by the Examiner and is unpersuasive of Examiner
error.
Appellant also argues that a person of ordinary skill in the art would
not have been motivated to combine Liu’s teachings of suggesting emoji
characters based on a user’s emotional state with the teachings of Radebaugh
Appeal 2019-005538
Application 15/347,653
15
and Grieves. Appeal Br. 26; Reply Br. 12–13. Particularly, Appellant
asserts that although Radebaugh identifies a challenge in converting text-to-
speech is conveying emotion or audible expressivity (Radebaugh ¶ 4),
Radebaugh describes a proposed solution by examining textual input for
emoticons (Radebaugh ¶ 5) and, thus, one of ordinary skill in the art would
not turn to Liu’s approach of sensing a user’s emotional state to suggest
emoji objects to assist in interpreting textual input. Reply Br. 12–13.
We are unpersuaded of Examiner error. As discussed above, the
Examiner determines that an ordinarily skilled artisan would have been
motivated to modify the expressive keyboard suggested by Radebaugh to
further include the emojis provided by the keyboard of Grieves “to reduce
difficulty for users to use emoji” instead of, or in addition to, the emoticons
of Radebaugh. Final Act. 6 (citing Grieves ¶ 3); Ans. 7. The Examiner
further finds one of ordinary skill in the art would have been motivated to
combine Liu’s teachings of mapping and presenting emoji characters to a
user’s perceived emotional state with the expressive keyboard taught by the
combined teachings and suggestions of Radebaugh and Grieves “to assist
with interpretation of the text.” Final Act. 12. We agree that incorporating
Liu’s teaching of presenting emoji characters based on a perceived
emotional state of the user with Radebaugh’s text-to-speech system, as
modified by Grieves, would assist in the interpretation of the text and further
reduce the difficulty in using emojis. See KSR, 550 U.S. at 418 (explaining
an obviousness analysis can “take account of the inferences and creative
steps that a person of ordinary skill in the art would employ”).
Accordingly, we sustain the Examiner’s rejection of claim 5.

Appeal 2019-005538
Application 15/347,653
16
CONCLUSION
We affirm the Examiner’s decision rejecting claims 1–4, 6–8, 13–15,
and 18 under 35 U.S.C. § 103.
We affirm the Examiner’s decision rejecting claim 5 under 35 U.S.C.
§ 103.
DECISION SUMMARY
Claims
Rejected
35 U.S.C. § Reference(s)/Basis Affirmed Reversed
1–4, 6–8,
13–15, 18
103 Radebaugh,
Grieves
1–4, 6–8,
13–15, 18

5 103 Radebaugh,
Grieves, Liu
5
Overall
Outcome
1–8, 13–
15, 18

TIME PERIOD FOR RESPONSE
No time period for taking any subsequent action in connection with
this appeal may be extended under 37 C.F.R. § 1.136(a). See 37 C.F.R.
§ 41.50(f).

AFFIRMED