TrackCap: Enabling Smartphones for 3D Interaction
on Mobile Head-Mounted Displays

Conference on Human Factors in Computing Systems (CHI 2019)

Peter Mohr
Graz University of Technology, VRVis GmbH
Markus Tatzgern
Salzburg University of Appkied Sciences
Tobias Langlotz
University of Otago
Andreas Lang
Graz University of Technology
Dieter Schmalstieg
Graz University of Technology
Denis Kalkofen
Graz University of Technology
Teaser: Our system enables using smartphones for 3D interactions in applications on head mounted displays (HMD). It consists of a planar tracking target attached to the HMD, a tracking software on the smartphone that estimates the pose of the phone relative to the HMD, and a network interface that transmits the pose to the HMD. This system enables interactions of the phone with virtual objects that are placed in world coordinates. In this example, we interact with virtual objects by attaching a smartphone to a plastic toy gun. While previous approaches require holding the device in the field of view of the HMD's camera (black), our system supports a large interaction space by using the camera on the phone for tracking (blue).
Link to ACM DL>
ACM
Link to Author Preprint
Paper Preprint
Link to Technical Video
Video

Abstract

The latest generation of consumer market Head-mounted displays (HMD) now include self-contained inside-out tracking of head motions, which makes them suitable for mobile applications. However, 3D tracking of input devices is either not included at all or requires to keep the device in sight, so that it can be observed from a sensor mounted on the HMD. Both approaches make natural interactions cumbersome in mobile applications. TrackCap, a novel approach for 3D tracking of input devices, turns a conventional smartphone into a precise 6DOF input device for an HMD user. The device can be conveniently operated both inside and outside the HMD's field of view, while it provides additional 2D input and output capabilities.

Overview

TrackCap enables new means of interaction. For example, rays for selecting objects can be quickly defined by swiping over the touchscreen (also see the accompanying video). (b) The high input and output fidelity of the smartphone can also be used to display detail of the selected objects and to enable high precision interactions with them TrackCap enables new means of interaction. For example, rays for selecting objects can be quickly defined by swiping over the touchscreen (also see the accompanying video). (b) The high input and output fidelity of the smartphone can also be used to display detail of the selected objects and to enable high precision interactions with them
Gamecontroller Setup
TrackCap gamecontroller
Tracked controller with high-resolution display

Citation 


@inproceedings{10.1145/3290605.3300815,
	author = {Mohr, Peter and Tatzgern, Markus and Langlotz, Tobias and Lang, Andreas and Schmalstieg, Dieter and Kalkofen, Denis},
	title = {TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays},
	year = {2019},
	isbn = {9781450359702},
	publisher = {Association for Computing Machinery},
	address = {New York, NY, USA},
	url = {https://doi.org/10.1145/3290605.3300815},
	doi = {10.1145/3290605.3300815},
	booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
	pages = {1–11},
	numpages = {11},
	keywords = {3d pointing, mixed reality, hmd, input devices, mobile devices, augmented reality, wearable computing},
	location = {Glasgow, Scotland Uk},
	series = {CHI '19}
	}