Tobias Schwandt - High-Quality Illumination of Virtual Objects Based on an Environment Estimation in Mixed Reality Applications

Responsible Editor: Stefanie Eggert

This Springer Vieweg imprint is published by the registered company Springer Fachmedien Wiesbaden GmbH part of Springer Nature.

The registered company address is: Abraham-Lincoln-Str. 46, 65189 Wiesbaden, Germany

This thesis has been created over a long period on a long road. On this way, different people have made an important contribution to itdirectly and indirectly. This includes intensive discussions and feedback as well as general support during difficult parts while writing the thesis. I would like to briefly mention a few people who have had a great impact on the approaches in this thesis.

First, I would like to mention Wolfgang Broll as my supervisor, professor at the Ilmenau University of Technology and head of the group for Virtual Worlds and Digital Games. A few years ago, he hired me and gave me the opportunity to start my thesis. During my time at the research group, I was constantly supported to define my research field. The time was characterized by regular meetings to discuss the current state of research and how to continue. He was always open-minded for new and innovative topics with constructive criticism of the work, the publications, and the next steps. Even besides regular meetings, the office door was always open. Many thanks Wolfgang for your support.

In addition to Wolfgang Broll, my colleagues Florian Weidner and Christian Kunert were outstanding companions. Both are also research assistants at the research group and pursue the goal of the thesis. Together we talked a lot about our research, exchanged new information and supported each other. We wrote several papers and defined ourselves in the research world. It is nice to be able to say that a private friendship has developed beyond work, which will hopefully continue for many years and many more papers. Florian and Christianthank you for your feedback, hints, support, and for the heartfelt laughter we had together.

Moreover, I would like to thank two other people who played an important role: my second reviewer Beat Brderlin from the Ilmenau University of Technology and the third reviewer, Thorsten Grosch from the Clausthal University of Technology. Collaboration and meetings were helpful and always productive. Thank you for supporting this thesis and the research.

On my way to the final submission, other people also contributed an important part, which does not concern the content. At this point I would like to thank my family and in particular, my wife Karoline. She is always by my side and accompanied me on sometimes winding streets. Since she is a research assistant herself, we were able to complement and support each other well. I love you!

I am pleased that this first big step in the research world was taken together.

Thank you.

Visualizations of virtual objects in the real environment is often done by a simplified representation with simple surfaces and without reference to the surrounding environment. The seamless fusion of the virtual and real environment is, however, an essential factor in many areas, which is of particular importance when calculating lighting in mixed realities on mobile devices. Current approaches focus on approximations, which allow the calculation of diffuse lighting, whereby the rendering of glossy reflection properties is often neglected.

The aim of this thesis is to enable the visualization of mirror-like reflective surfaces in mixed reality. In order to achieve this goal, various approaches are explored enabling high-quality visualization of virtual objects in realtime with a focus on the use of common hardware such as cameras, sensors in mobile devices, and partially depth sensors. The first approach uses the current camera image to reconstruct 360-degree lighting by transforming the current camera image directly onto a cubemap in each frame. On top of this approach, the technique is enhanced by stitching the current camera image in combination with the position and rotation to estimate a 360-degree lighting. Uncaptured parts by the camera are filled with semantic inpainting using a neural network. In this way, complete ambient lighting can be estimated, which enables detailed reflections. The results provide a novel way to embed complex and simple geometric shapes with glossy surfaces in the real world which, in contrast to existing approaches, offers a higher level of detail in the reflections without using additional hardware. All approaches are designed to be used on mobile devices which provides new opportunities for the different use cases and areas of application for modern mixed reality.

Die Visualisierung virtueller Objekte in der Realitt erfolgt bei vielen Anwendungen oftmals durch eine vereinfachte Darstellung ohne Bezug zur umliegenden Umgebung. Dabei ist die nahtlose Verschmelzung der virtuellen und realen Umgebung in vielen Bereichen ein wesentlicher Faktor, der insbesondere bei der Beleuchtungsberechnung in gemischten Realitten von groer Bedeutung ist. Aktuelle Anstze legen den Fokus auf Approximationen, welche eine Berechnung der diffusen Beleuchtung ermglichen, wobei die Darstellung glnzender Beleuchtungseigenschaften vernachlssigt wird.