MR bare-hand
Seamless Bare-Hand Interaction in Mixed Reality
Goal of this internship was to help to create a user appropriate, contextually aware AR system, through a seamless integration of core technologies and applications on a state-of-the-art mobile platform.
Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualization when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualization of the hand in MR. We propose an approach that automatically localizes the hand in RGB images and associates the respective depth, estimated with an auxiliary infrared stereo camera used for hand tracking, to each RGB hand pixel.
Because misalignment between the outline of the hand and its depth may occur due to tracking errors, we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared, and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it.
Bare-hand interaction with the augmentation of a hat and a bunny using our proposed approach. When the depth of the hand is processed and associated to the pixels labelled as hand, object rendering can account for occlusions thus enabling seamless MR experiences.
Publication
C. Battisti, S. Messelodi, F. Poiesi, Seamless Bare-Hand Interaction in Mixed Reality, IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Munich, Germany, October 2018, pp. 198-203
Work performed from February 2018 to July 2018
Student: Caterina Battisti (Dep. of Information Engineering and Comp. Science, UniTN)
Supervised by: Fabio Poiesi, Stefano Messelodi (FBK)
Caterina attended ISMAR 2018, this is her experience.
My name is Caterina and ISMAR 2018 has been the first scientific conference I ever attended to. The opportunity to show and explain the work of my 3-months internship at FBK, “Seeamless bare-hand interaction in Mixed Reality”, to the AR/VR/MR community has been an invaluable experience for me. During the three days of conference, I was able to attend several paper presentations regarding the Mixed/Augmented reality world. The experts presenting their work were really professional and inspiring and I feel like I learned a lot from them. I also had the chance to step on stage and pitch my work. A very short time to catch the audience attention and convince them to check out my poster. I was really excited about seeing how many people were interested in hearing about the details of how I achieved the results of my work. I was able to collect opinions of people from all around the world, see their point of view, their impressions and reactions to my work. The most precious moment was seeing the amazed looks of people trying out the simple demo I prepared and set up next to the poster. Even coffee breaks have been an important part of the experience. I was able to talk to researchers and experts and discuss about how they got so far in their career. This was of great inspiration for my own professional career as I was constantly thinking about my future and how much is waiting for me. This experience as a whole was a deep dive into research, future, innovation, inspiration.
Font: https://women.acm.org/scholars/acm-w-scholars/caterina-battisti/