CVPR 2015 Tutorial: Applied math as applied in cinema

2PM to 6PM on Sunday June 7, 2015. Room 105, Hynes Convention Center. Boston, Massachusetts.


This tutorial covers a wide range of topics showing how applied math techniques from image processing and computer vision have become ubiquitous in movie-making, from shooting to exhibition. It does not deal with visual effects or computer-generated images, but rather with all the ways in which applied math is used to enhance, restore, adapt or convert moving images, the purpose of these techniques being to make the images look as good as possible while exploiting all the capabilities of cameras, projectors and displays.

Current digital cinema cameras match or even surpass film cameras in color capabilities, dynamic range and resolution, and several of the largest camera makers have ceased production of film cameras. On the exhibition side, film is already virtually gone from American movie theaters. And while many mainstream movies are still being shot on film, they are all digitized for postproduction. Therefore, in this tutorial we will equate ``cinema'' with ``digital cinema,'' considering only digital cameras and digital movies, and not discussing algorithms for problems that are inherent to film, like the restoration of film scratches or color fading.

Despite the “applied math” category, very few of the algorithms intended for application in the cinema industry are ever actually used. While there are many reasons for this, it is often the case that researchers in academia are not fully aware of the impossibly high quality standards of cinema, also lacking a clear picture of what the actual needs of the industry are. This tutorial addresses these issues head-on, providing a detailed overview of several very relevant problems in cinema production, postproduction and exhibition, and the way computer vision methods can be used to solve them.


Topics covered and brief outline:

  • In-camera image processing: Image processing pipeline, Image sensors, Exposure control, Focus control, White balance, Color transformation, Gamma correction and quantization, Edge enhancement, Output formats

  • Noise and dynamic range: Classic denoising ideas, Non-local approaches, New trends and optimal denoising, High dynamic range imaging, Tone mapping

  • Color correction: Human color constancy, Computational color constancy under uniform illumination, Retinex and related methods, Cinema and colors at night, Color matching, Color stabilization

  • Image stabilization: Rolling shutter compensation, Compensation of camera motion

  • Zoom-In and Slow Motion

  • Gamut mapping: Color gamuts, Gamut reduction, Gamut extension,Validating a gamut mapping algorithm

  • Inpainting: Video inpainting for specific problems, Video inpainting in a general setting, Video inpainting for stereoscopic 3D cinema



The tutorial follows the book “Image Processing for Cinema” by M. Bertalmío, published in 2014 by CRC Press. Some reviews:

"… it is a great relief to find Image Processing for Cinema by Marcelo Bertalmío. He wrote it as a textbook for graduate students in areas including applied mathematics, image processing, and computer science, as a comprehensive guide to digital cinema for industry professionals. … Bertalmío’s explanations are in a league with Walter (The Technique of the Film Cutting Room)—written in a way that doesn’t cause the eyes to glaze over. It is technical writing that we can read. It takes the reader into a subject but gives him or her exit points at various levels of curiosity satisfaction. … this book is a resource of in-depth information on digital image technology."
—Jay Cassidy, 2014, 2013, and 2008 Academy Award nominee for best achievement in film editing.
CINEMAEDITOR, Vol. 64, 2014.

"This is a comprehensive, informative, and well written book, which covers all aspects of the production and processing of cinema. (Cinema means digital cinema here.) I have not seen any other text that serves this purpose. It does not deal with visual effects or computer generated images, but with the ways algorithms are used to make images look as good as possible. The author attempts the Herculean task of going from light and color through optics to how cameras work and the image processing algorithms used in-camera. Finally, offline image processing algorithms are discussed. The goal is to bridge the communication gap between movie professionals and image processing researchers. There are enough equations to satisfy most researchers but the material remains accessible to nonexperts. There is a lot of material on open questions (like how humans really perceive color), which could provoke new research. I recommend this book strongly. It is suitable for people from the movie industry to researchers as well as undergraduate and graduate students interested in imaging and cinema."
—Stanley Osher, Professor of Mathematics, Computer Science, and Electrical Engineering, University of California, Los Angeles



Marcelo Bertalmío. Associate Professor, Universitat Pompeu Fabra, Spain.

Short bio:

Marcelo Bertalmío (Montevideo, 1972) received the B.Sc. and M.Sc. degrees in electrical engineering from the Universidad de la República, Uruguay, and the Ph.D. degree in electrical and computer engineering from the University of Minnesota in 2001. Since 2006 he is an Associate Professor at Universitat Pompeu Fabra, Spain.

His publications total more than 7,000 citations. He was awarded the 2012 SIAG/IS Prize of the Society for Industrial and Applied Mathematics of the USA (SIAM) for co-authoring the most relevant image processing work published in the period 2008-2012. Has received the Femlab Prize, the Siemens Best Paper Award, the Ramón y Cajal Fellowship, and the ICREA Academia Award, among other honors. He is an Associate Editor for SIAM-SIIMS and the secretary of SIAM's activity group on imaging. Has an ERC-Starting Grant for his project “Image processing for enhanced cinematography”. Has written a book titled “Image Processing for Cinema”, published by CRC Press / Taylor & Francis.

His current research interests are in developing image processing algorithms allowing to shoot cinema with no more artificial lighting than what people present at the scene need to be able to see. The approach is to work out software methods mimicking neural processes in the human visual system, and apply them to images captured with a regular digital movie camera.