A window to your smartphone: exploring interaction and communication in immersive VR with augmented virtuality

Abstract

A major drawback of most Head Mounted Displays (HMDs) used in immersive Virtual Reality (VR) is the visual and social isolation of users from their real-world surroundings while wearing these headsets. This partial isolation of users from the real-world might hinder social interactions with friends and family. To address this issue, we present a new method to allow people wearing VR HMDs to use their smartphones without removing their HMDs. To do this, we augment the scene inside the VR HMD with a view of the user's device so that the user can interact with the device without removing the headset. The idea involves the use of additional cameras, such as the Leap Motion device or a high-resolution RGB camera to capture the user's real-world surrounding and augment the virtual world with the content displayed on the smartphone screen. This setup allows VR users to have a window to their smartphone from within the virtual world and a�ord much of the functionality provided by their smartphones, with the potential to reduce undesirable visual and social isolation users may experience when using immersive VR HMDs. This work has been successfully submitted for presentation as a poster in the Computer and Robot Vision Conference 2017 in Edmonton, Alberta, and is scheduled to appear in the conference proceedings at the IEEE Xplore digital library later this year.