Artusi Alessandro - Image Content Retargeting: Maintaining Color, Tone, and Spatial Consistency

IMAGE CONTENT RETARGETING

Maintaining Color, Tone, and Spatial Consistency

IMAGE CONTENT RETARGETING

Maintaining Color, Tone, and Spatial Consistency

Alessandro Artusi

Universitat de Girona, Spain

Tun Ozan Aydn

Disney Research, Zrich, Switzerland

Olga Sorkine-Hornung

ETH Zrich, Switzerland

Francesco Banterle

ISTI-CNR, Pisa, Italy

Daniele Panozzo

ETH Zrich, Switzerland

CRC Press

Taylor & Francis Group

6000 Broken Sound Parkway NW, Suite 300

Boca Raton, FL 33487-2742

2017 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Printed on acid-free paper

Version Date: 20160607

International Standard Book Number-13: 978-1-4822-4991-0 (Hardback)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Visit the Taylor & Francis Web site at

http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com

[To all my family, always in my heart. - AA]

[To my nieces. - FB]

Contents

Foreword

When I was growing up in Spain, many more years ago than Id like to admit, there was only one TV in my house. A black and white TV, with exactly two channels. They didnt really turn their brains into mush naming them: They were called Channel 1 and Channel 2. To be more precise, it was more like one and a half channels, since Channel 2 would only emit content from 7 PM until midnight. My favorite cartoons would be shown exactly once a week. And because recording devices were an outrageously expensive luxury item back then, if you missed them, they were gone. Forever. You would never, ever, be able to watch them again.

And boy, it was fantastic!

It didnt matter that I had to imagine the colors of Mazinger Z and its robot enemies from the gray-scale images on the TV. It also didnt matter that when an American movie was shown, either big chunks at the top and bottom of the screen would be pure black, or that the images would be badly distorted, in an effort to accommodate the different aspect ratios of the movie and the TV set. I didnt care about dynamic range, gamut mapping, color-to-gray conversions, all of which, of course, I had no idea they were even a thing. I had access to visual content, and that was cool!

Now times have changed. Images and movies are captured from many devices, and can be shown on many, many different displays. In color! We have large, 55-inch (and more!) screens; we have I-Max theaters; we have computer monitors, cellphones, tablets, projectors. We have canned and streamed video. We have a gazillion formats and compression algorithms to store all kinds of visual media. Color is not just color; we have many, many color appearance models. The combinatorial space of how my favorite childhood cartoons can be displayed is virtually infinite. But hold on a sec: How can we show the same content to different people with so many different display devices?

This is the process called content retargeting. And this book tells you how it is done. You will first learn about the key aspects of our visual perception that need to be taken into account when mapping visual media to different displays. Then color and dynamic range will be introduced; quoting my friend Erik Reinhard, currently with Technicolor, both can be considered a two sides of the same coin, so they are presented together for a proper, holistic view. How to turn a color image into a gray-scale image is the topic of the following chapter. Color is a higher-dimensional space, so some information needs to be discarded, and other information transformed. Now how, that is the issue. But things get even more interesting when you consider the role of style, and ask yourself the question of how to transfer it from one image to another. I have this great picture I took on holidays, but wouldnt it be great if I could somehow make it look a bit more like this other picture from this famous photographer? Maybe you have a clear mental depiction of the desired result, but unfortunately not all of us are that good at using commercial image manipulation software like Photoshop. What we want is to simply tell the computer, and let it produce the result automatically. Great, but how? Finally, the book deals with spatial retargeting, or how to accommodate visual media on the different sizes and aspect ratios of the many, many screens available to us. This has been a key research area in the past few years, with many works trying to puzzle this out. And finally, after everything is said and done, we are left with the question of OK, but how do we know if we have done a good job? How do we know if the resulting image, after all our transformations in many different spaces, is acceptable to a human observer? Enter the domain of image quality assessment, which will be thoroughly analyzed in the last chapter of the book.

All in all, this book is as timely as it gets, given the massive, massive amount of available visual media and the smorgasbord of display devices of very different characteristics that exist nowadays. It provides a cross section of what is possible today in image content retargeting, and a window to what may be possible tomorrow. If only I had known as a kid, I might have appreciated my Saturday morning Mazinger-Z shows even more.

Diego Gutierrez
September 2015
Zaragoza University

Preface

The availability on the market of a large variety of color display devices, characterized by significant differences in their color gamut, dynamic range, spatial resolution, etc., has brought a series of research challenges that have been a topic of discussion in the last 30 years. A typical problem is the acquisition of an image with a digital camera and its subsequent visualization on various media and display devices with different characteristics. If the acquired image is not processed in advance, the user experience will be to view the image on a display medium that may have large discrepancies in its colors and tone, dynamic range, and even image structure. Sometimes the color information of the image is discarded due to the constraints imposed by display devices with limited capability, i.e., gray-level media. The question is, therefore, how do we preprocess images to maximize the display quality on the particular device? Another classical problem is the need to transfer a specific set of colors, as well as the style, from one image to another one. The research problems and applications above can be classified as