You are here

Sensor-based Gesture and Activity Recognition

Our activities focus on signal processing techniques for detecting user-customized gestures as well as recognising repetitive motion patterns such as occurring during work-out and sport activities.

Novel portable appliances such as smart-watch and smart-phone are equipped with inertial sensors that can be used to measure movement. Our activities focus on signal processing techniques for detecting user-customized gestures as well as recognising repetitive motion patterns such as occurring during work-out and sport activities.

Modern mobile devices provide several functionalities and new ones are being added at a breakneck pace. Unfortunately browsing the menu and accessing the functions of a mobile phone is not a trivial task for visual impaired users. Low vision people typically rely on screen readers and voice commands. However, depending on the situations, screen readers are not ideal because blind people may need their hearing for safety, and automatic recognition of voice commands is challenging in noisy environments. Novel smart watches technologies provides an interesting opportunity to design new forms of user interaction with mobile phones.

Our first work towards the realization of a system is based on the combination of a mobile phone and a smart watch for gesture control, for assisting low vision people during daily life activities. Our approach for gesture recognition is based on global alignment kernels.  It is shown to be effective in the challenging scenario of user independent recognition. This method is used to build a gesture-based user interaction module and is embedded into a system targeted to visually impaired which can integrate several computer vision modules, like the detections of signs or logos.

Publications:

L. Porzi and S. Messelodi and C.M. Modena and E. Ricci: A Smart Watch-based Gesture Recognition System for Assisting People with Visual Impairments. ACM International Workshop on Interactive Multimedia on Mobile and Portable Devices  - IMMPD, Barcelona, Spain, October 2013

G. Costante, L. Porzi, O. Lanz, P. Valigi, E. Ricci: Personalizing a Smartwatch-based Gesture Interface with Transfer Learning. European Signal Processing Conference - EUSIPCO, Lisbon, Portugal, September 2014