PATENTSCOPE will be unavailable a few hours for maintenance reason on Tuesday 19.11.2019 at 4:00 PM CET
Search International and National Patent Collections
Some content of this application is unavailable at the moment.
If this situation persists, please contact us atFeedback&Contact
Note: Text based on automatic Optical Character Recognition processes. Please use the PDF version for legal matters



The present invention relates to a method, device and a computer program for controlling input of non-textual symbols in a device, and especially in a communication device.


Mobile telephones have evolved during the last few years from being simple voice communication devices to present day intelligent communication devices having processing and communication capabilities. The use of a mobile telephone may involve such activities as interactive messaging, sending e-mail messages, browsing the World Wide Web and many other activities, both business related as well as more leisure oriented. Moreover, the operation of current communication devices is often controlled via user interface means that include, in addition to or instead of traditional keypads, touch sensitive displays on which a virtual keypad is displayed. In the latter case, usually a user inputs text and other symbols using a stylus by touching the virtual keypad.

Instant messaging and chat is very popular and one important part is to express emotions using smileys, keyboard character combinations mapped to emoticons.

Originally the smileys were formed as plain text characters, like :-) and ;(. However, in current messaging and chatting applications, smileys are also provided as unique nontextual symbols, which are small graphical bitmaps.

A drawback with current devices, such as mobile phones, PDAs, etc., is that they typically have to display a plurality of possible non-textuai symbols, including the smileys, for selection by the user. A user must usually select from a list of smileys or use symbols to form a smiley, which depending on applications may be converted to a graphical smiley. When chatting, for example, this may be problem as the user must choose from a list and find a correct smiley. This is time consuming and may need a number of additional moments.


The modern mobile telephones, computers, PDAs and communication devices usually comprise one or several image recording devices in form of cameras. Especially, mobile telephones enabled for video calls have a camera directed towards the user. The present invention uses the advantage of having a camera on a messaging device such as a mobile telephone to generate symbols, preferably non-textual symbols such as smileys. Thus, the proposed solution uses face detection preferably combined with facial part analysis to automatically generate the emoticons.

Thus, the invention according to a first aspect relates to a method for inserting non-textual information in a set of information. The method comprises the steps of: using a facial image of a user, generating a first data set corresponding to the facial image, comparing the first data set with a stored data set corresponding to the non-textual information, selecting a second data set based on the comparison, and providing the second data set as the non-textual information into the set of information. Preferably, the set of information is a text based information. The set of non-textual information is an emoticon. Preferably, the emoticon corresponds to the facial appearance of the user.

The invention also relates to a device according to a second aspect comprising a processing unit, a memory unit and an image recording arrangement. The image recording arrangement is configured to capture at least a portion of a users face, the processing unit is configured to process the captured image corresponding to at least the portion of the users face and compare it to a data set stored in the memory, the processing unit being further configured to select a data set based on the comparison. The selected data is output to a text processing unit. The device may further comprise a display, input and output units, a transceiver portion and an antenna.

The invention also relates to a computer program stored on a computer readable medium for inserting non-textual information in a set of information. The computer program comprises: a set of instructions for selecting a facial image of a user, a set of instructions for generating a first data set corresponding to the facial image, a set of instructions for comparing the first data set with a stored data set corresponding to the non-textual information, a set of instructions for selecting a second data set based on the comparison, and a set of instructions for providing the second data set as the non-textual information into the set of information.

BRIEF DESCRIPTION OF THE DRAWINGS In the following, the invention is described with reference to drawings illustrating some exemplary embodiments, in which:

Fig. 1 shows a schematically drawn block diagram of an embodiment of a mobile communication device according to the invention, Figs. 2a-2c show schematically block diagram of a facial recognition embodiment according to the invention, Fig. 3 is a flow diagram illustrating exemplary method steps according to the present invention.

Fig. 4 is a schematically drawn block diagram of an embodiment and screen shots of a communication device during execution of a computer program that implements the method of the present invention.


Fig. 1 illustrates schematically a communication device in the form of a mobile telephone device 100. The device 100 comprises a processor 101 , memory 102, one or several cameras 103, a display 104, input and output units 105, a transceiver portion 106 and an antenna 107. The display may be a touch sensitive display on which a user writes using, e.g., a stylus or similar device. Other input/output units 105 in the form of a speaker, a microphone, a keyboard may also be present, functions of which are well known for a skilled person and not described herein in detail. The display 103, input/output units 105 and the camera 103 may communicate with the processor 104 through an l/0-interface (not shown). The details regarding how these units communicate are known to the skilled person and are therefore not discussed further. The communication device 100 may, in addition to the illustrated mobile telephone device, be a Personal Digital Assistant (PDA) equipped with radio communication means or a computer, stationary or laptop equipped with a camera.

The telephone 100 is capable of communication via a transceiver unit 106 and an antenna 107 through an air interface with a mobile (radio) communication system (not shown) such as the well known systems GSM/GPRS, UMTS, CDMA 2000 etc.

The present invention uses one of the mobile phone's sensors, preferably the video telephony camera 103, to automatically generate emoticons (smileys) contrary to the classic input methods using the keyboard or touch screen display to enter the right character combinations.

Figs. 2a-2c and 3 in conjunction with Fig. 1 , illustrate the principles of the invention according to one embodiment. When an application, such as chatting or text processing application, with ability to use smileys is started (1 ) a user's face 250a-250c (happy, blinking and unhappy, respectively) is captured (2) by the camera 103 of the exemplary communication device. A facial recognition portion 1 10, implemented as hardware or an instruction set executed by the processor 101 processes (3) the recorded image from the camera and searches for certain characteristics, such as leaps (motion), eyes, cheeks etc., and the processor 101 looks up (4) for the similarities, e.g. in a look up table in the memory 102. When a smiley or emoticon similar to facial recognized data is found and selected (5), it is outputted (6) as smileys 255a-255c (smiling/happy, wink and frowning/sad, respectively) into the application 260 which calls the functionality of the present invention. The procedure is executed until (7) the application is terminated or the user decides to use other input means, for example.

The smileys or emoticons may be in form of so-called western style, eastern style, East Asian Style, ideographic style, a mixture of styles or any other usable styles.

One benefit of the proposed solution is that the user can interact using his/her face via the camera to express emotions in text form.

Fig. 4 illustrates an exemplary application embodiment during an instant messaging chat session:

1 ) The user 250 writes a text message 520 and during writing the video telephony camera of the mobile phone 100 analysis the facial parts to find out when the user wants to express an emotion in the text 521.

2) If the user winks with the eye, a wink smiley 522 is automatically generated at the current text cursor position.

3) If the user smiles to express happiness, a happy smiley 523 is automatically generated at the current text cursor position.

The method according to the preferred embodiments will in general reside in the form of software instructions of a computer program with an associated memory area 1 10, together with other software components necessary for the operation of the device 100, in the memory 102 of the device 100. The computer program 110 may be resident or it may be loaded into the memory 102 from a software provider, e.g. via the air interface and the network, by way of methods known to the skilled person. The program will be executed by the processor 101 , which will receive and process input data from the camera and input units, keyboard or touch sensitive display (virtual keyboard) in the device 100

It should be noted that the word "comprising" does not exclude the presence of other elements or steps than those listed and the words "a" or "an" preceding an element do not exclude the presence of a plurality of such elements. It should further be noted that any reference signs do not limit the scope of the claims, that the invention may be implemented at least in part by means of both hardware and software, and that several "means", "units" or "devices" may be represented by the same item of hardware.

The above mentioned and described embodiments are only given as examples and should not be limiting to the present invention. Other solutions, uses, objectives, and functions within the scope of the invention as claimed in the below described patent claims should be apparent for the person skilled in the art.