Universal Mobile Conduit for Cyber Physical Assisted Living
Principal Investigators: Young H. Cho (youngcho@isi.edu) and Jihie Kim (jihie@isi.edu)
Research Engineer: Dongyoul Lee (elecs4@gmail.com)
University of Southern California – Information Sciences Institute
Marina del Rey, CA 90292


    Today, the most serious problem that the older residents of assisted living facility (ALF) have is not a particular physical illness. Studies discovered that most residents struggle psychologically with the feeling of loneliness which often leads to depression and other related physical and mental illness. With the projection that the traditional retirement age group will grow to be the largest group of people in the United States by 2030, as many as 35% of the entire working population (U.S. Census Bureau), an efficient and maintainable solution should be implemented. We believe that there must be a concerted effort to extend and improve the psychological health of older population through seamless integration of innovative computing and social networking technologies into their daily lives. Our unique approach for solving this problem is in an attempt to merge and expand the reach of cyberspace and physical space communities by designing technologies that will allow seamless interaction between the two with minimal effort. We plan to develop (a) an advanced user interface that reduces barriers to digital communication, and (b) a novel social networking environment that brings older individuals closer to others by promoting social interactions in the facility and strengthens connections to family and friends outside the facility.
    As a part of (a), we have begun developing Universal mobile conduit (UMC) system to conduct feasibility study on the natural user interface for the elderly.  Since there is a wide range of user capacity in elderly community, we intend to integrate a wide range of user interface methods that can be selected. Our current design of UMC is developed on Android based Google phones to enable natural user interface for elderly residents of Assisted Living Facilities (ALF). The system requires programs running on the mobile phone working as a direct interface for the users and a driver application running on a PC to emulate the user input. The application is designed to replace traditional user input devices such as mouse and keyboard. The sensors in the mobile phone are used to detect the movement and the gestures of the phone for desired control of the mouse pointer while the voice enable keyboard mechanism is used to emulate key strokes on the PC.

I.    Overview

The current version of UMC emulates a number of different wireless user interface.  In terms of pointer control, the touch screen of the phone can be configured to be used as mousepad.  Although such implementation already exists, the main contribution during the development of this functionality is that UDP is used as the main communication protocol to transfer control data between the programs while Bluetooth is used as the means to detect proximity of the mobile phone and PC for automatic IP assignment and detection.  This design allows the user interface to be used in more diverse computing environment.  Under this communication framework, we also implemented a phone movement based pointer control which may offer better option for some seniors with decrease control over finger movements.  In terms of keyboard input, we have implemented two methods of input.  The first method is through simple use of built-in keyboard on the phone (either touch screen or a pull out mini-keypad).  Again, for others, we have imntegrated voice recognition based keyboard function.

II.    Remote Touchpad and Keyboard

First function of the UMC is touchpad mouse. The application maps a large part of the touch screen to determine the positions of the finger on the screen.  Then the phone computes the relative position of the mouse on the PC using historical information and the display. In addition to the touch pad emulation for the pointer movement, the rest of the the touch screen is configured to act like touchpad buttons.  Keypad of the phone is programmed to emulate keyboard on PC (shown in Figure 1).

Figure 1: Google Phone based Mousepad and Keyboard

The data produced from the touch screen and the keypad are packaged as UDP packets and send via wireless Ethernet connection to the corresponding IP address of the target PC.

III.    Remote Motion Pointer

Second function of the UMC is motion sensing pointer controller.  This function is made possible through the use of the 3-axis accelerometer/orientation sensor of the mobile phone. Given the access to real-time data from orientation sensors, we implemented a simple algorithm to convert the phone to function like a Nintendo Wii game controller as indicated in Figure 2.

Figure 2: Nintendo Wii Controller like function with added gesture recognition for emulating selecting and clicking of left and right mouse key

During the development time of this simple function, we found that the sensor outputs were too sensitive to be directly used in controlling the mouse movement.  Therefore, we implemented a simple low-pass filter to remove the noise which appeared as shaky movement of the pointer.

IV.    Voice enabled keyboard

Current version of UMC also supports the voice recognition based keyboard input. Although it is possible to integrate and run voice recognition software on the phone itself, we decided against it due to our lack of expertise in that area as well as limited computing resource on the phone.  Instead, we implemented our system using Android's voice enabled keyboard functions.  The function allows the recorded voice to be sent to the remote Google server to process and return recognized text back to the phone as shown in Figure 3.  We simply had to write a glue logic to transfer the text to the PC application.  This function will be integrated in to a system that will effectively issue voice commands for supporting other users with even less motor control.

Figure 3: Google voice enabled keyboard based user interface

V.    PC Proximity Detection

In order to simplify configuration process, the application uses Bluetooth signal strength service to automatically detect IP address of PC that is closest to the phone. This mechanism not only allows seamless interaction between the Android phone and the PC, it also automatically enables/disables their interactions based on the proximity of the phone and the PCs.

VI.    Mouse emulation driver program on PC

The application of pc-side functions as server to emulate mouse and keyboard.  In order to receive the values corresponding to the coordinate of mouse pointer, UDP packets with the sensor data are sent via wireless network interface. The Robot class of java is used make the driver for the mouse and keyboard emulation on the PC.

VII.    The List of Available Command Interface

PRESS_MOUSE(-1) : When pressing mouse button
RELEASE_MOUSE(-2) : When releasing mouse button
PRESS_KEY(-3) : When pushing key boards
RELEASE_KEY(-4) : When releasing key boards
MOVE_MOUSE(-5) : When moving mouse by phone.
TOUCH_DOWN(-6) : When starting touch on phone screen
SENSING_START(-7) : When starting to use the orientation sensor
COORD_RESPOND(-100) : To send XY coordination of mouse of PC to mobile
IP_REQUEST(-1000) : To request IP address of PC from mobile phone
IP_RESPOND(-1001) : To respond IP address of PC to mobile phone

VIII.    Download Application

Mobile APP For Client - This software can be downloaded and installed via web browser of the phone

PC APP For Server - This software can be downloaded on to the PC with Windows OS with java runtime engine