Neural Cameras: Learning Camera Characteristics for Coherent Mixed Reality Rendering

IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR)

Best Conference Paper

David Mandl
Graz University of Technology
Peter Mohr
Graz University of Technology
Tobias Langlotz
University of Otago
Christoph Ebner
Graz University of Technology
Shohei Mori
Graz University of Technology
Stefanie Zollmann
University of Otago
Peter M. Roth
Technical University of Munich
Denis Kalkofen
Graz University of Technology
Teaser image


Coherent rendering is important for generating plausible Mixed Reality presentations of virtual objects within a user’s real-world environment. Besides photo-realistic rendering and correct lighting, visual coherence requires simulating the imaging system that is used to capture the real environment. While existing approaches either focus on a specific camera or a specific component of the imaging system, we introduce Neural Cameras, the first approach that jointly simulates all major components of an arbitrary modern camera using neural networks. Our system allows for adding new cameras to the framework by learning the visual properties from a database of images that has been captured using the physical camera. We present qualitative and quantitative results and discuss future direction for research that emerge from using Neural Cameras.


	author={Mandl, David and Roth, Peter M. and Langlotz, Tobias and Ebner, Christoph and Mori, Shohei and Zollmann, Stefanie and Mohr, Peter and Kalkofen, Denis},
	booktitle={2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)}, 
	title={Neural Cameras: Learning Camera Characteristics for Coherent Mixed Reality Rendering},