Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk,

Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk,

Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel INFORMATIK High Dynamic Range Luminance [log cd/m^2] -6 -4 -2 0

2 CRT Monitor LCD Monitor Bright projector Human eye 4 6 8 High vs Low Dynamic Range Video

LDR Video HDR Video Intended for existing displays Intended for the human eye Relative pixel brightness Photometric or radiometric units [cd/m2, Watt/m2sr] High Dynamic Range Video Goal: Efficient encoding of full dynamic range of luminance perceived by the human observer

Luminance [log cd/m^2] -6 -4 -2 0 2 4 6 8

1st demo Overview 1. HDR Pipeline 2. HDR Video Encoding Luminance Quantization Edge Coding 3. Results vs. MPEG-4 vs. OpenEXR 4. Demo & Applications

Related Work HDR Pipeline Acquisition Storage Display Related Work HDR Pipeline Acquisition Storage Display Global Illumination HDR Cameras HDRC (IMS Chips) Lars III (Silicon Vision) Autobrite (SMal Camera Technologies) LM9628 (National) Digital Pixel System (Pixim) Technology overview [Nayar2003] HDRC IMS Chips

Related Work HDR Pipeline Acquisition Storage Display Still images Radiance RGBE [Ward91] OpenEXR [Bogart2003] logLuv TIFF [Ward98] HDR JPEG [Ward2004] Video No video format Related Work HDR Pipeline Acquisition Storage Display LDR Displays But Tone Mapping necessary HDR displays start to appear

University of British Columbia [Seetzen2004] HDR Encoding Framework Detail level 1: Input & Output LDR HDR Video encoder White: MPEG Orange: HDR Encoder bitstream HDR Encoding Framework Detail level 2: Color Transform

LDR HDR Color Transform White: MPEG Orange: HDR Encoder YCrCb Lp u'v' Video Encoder bitstream HDR Encoding Framework Detail level 3: Edge Coding

LDR HDR Color Tran. Motion Comp. White: MPEG Orange: HDR Encoder DCT Coding Variable length bitstream

Edge Coding Runlength Encoding of Color LDR RGB HDR XYZ Color Tran. Motion Comp. DCT Coding

Variable length bitstream Edge Coding Runlength Encoding of Color How to represent color data? Floating Points ineffective compression Integers ok, but require quantization How to quantize color data? Quantization errors < threshold of perception Use uniform color space (L*u*v*, L*a*b*) [Ward98]

Find minimum number of bits Color (u*v*) 8 bits are enough Encoding of Luminance How to quantize luminance? Gamma correction? Logarithm? 8 6 log(Y)? log Luminance Y 4 2 0

-2 -4 Integer representation Threshold Versus Intensity Psychophysical measurements The smallest perceivable difference Y for a certain adaptation level YA tvi [Ferwerda96, CIE 12/2.1] Y YA - Adaptation Luminance log Threshold Y 6

tvi 4 2 0 -2 -4 -2 0 2 4 6

log Adaptation Luminance YA 8 Luminance Quantization Just below threshold of perception 8 tvi(Y ) f 6 log Luminance Y Maximum quantization error

emax 4 2 0 -2 -4 0 Integers Lp 2048 Luminance Quantization Just below threshold of perception 8

d (l ) 2 f 1tvi( (l )) dl l L in LP space (l ) LP to Y mapping f threshold decrease tvi(Y ) f 6 log Luminance Y Maximum quantization error emax

4 2 0 -2 -4 10 11 bits are enough 0 Integers Lp 2048 Capacity function [Ashkihmin02] Grayscale Standard Display Function [DICOM03]

Luminance Quantizations Comparison log Contrast Threshold 2 cvi 11-bit percep. quant. 32-bit LogLuv RGBE 0 -2 -4 -4 -2

0 2 4 6 log Adapting Luminance 8 Edge Coding LDR RGB HDR XYZ Color Tran. Motion Comp.

DCT Coding Variable length bitstream Edge Coding Runlength Edge Coding: Motivation HDR video can contain sharp contrast edges Light sources, shadows DCT coding of sharp contrast may cause high frequency artifacts DCT coding

Edge coding Edge Coding: Solution Solution: Encode sharp edges in spatial domain, the rest in frequency domain Sharp edge signal Run-length encoding Original Signal Smoothed signal DCT encoding Edge Coding: Algorithm original

I horizontal decomposition edge block horiz. edges II horizontal DCT III vertical decomposition IV vertical DCT edge block

vert. edges Edge Coding: Algorithm original I horizontal decomposition edge block horiz. edges II horizontal DCT III vertical decomposition

IV vertical DCT edge block vert. edges Edge Coding: Algorithm original I horizontal decomposition edge block horiz. edges II horizontal DCT

III vertical decomposition IV vertical DCT edge block vert. edges Edge Coding: Algorithm original I horizontal decomposition edge block

horiz. edges II horizontal DCT III vertical decomposition IV vertical DCT edge block vert. edges Edge Coding: Algorithm original

I horizontal decomposition edge block horiz. edges II horizontal DCT III vertical decomposition IV vertical DCT

edge block vert. edges Results 2x size of tone-mapped MPEG-4 video 20-30x saving compared to intra-frame compression (OpenEXR) MPEG-4 HDR Enc. 0.55 1 27 OpenEXR Bit-stream Size Demo & Applications

Display dependent rendering Choice of tone-mapping Extended postprocessing Conclusions HDR video compression Modest changes to MPEG-4 Lpuv color space Luminance quantization (10-11 bits) Edge coding Applications On-the-fly tone mapping Blooming, motion blur, night vision Tuned for display LDR / HDR Display Acknowledgments

HDR Images and Sequences Paul Debevec SpheronVR Jozef Zajac Christian Fuchs Patrick Reuter HDR Camera HDRC(R) VGAx courtesy of IMS CHIPS www.hdrc.com

Comments and help Volker Blanz Scott Daly Michael Goesele Jeffrey Schoner Thank you http://www.mpi-sb.mpg.de/resources/hdrvideo/

Recently Viewed Presentations

  • Geometry Proofs - Weebly

    Geometry Proofs - Weebly

    Geometry Proofs Obj: To Complete Proofs Involving Segment Theorems ... and transitive Angles supplementary to the same angle or to congruent angles are congruent Theorems Continued Angles complementary to the same angle or congruent angles are congruent All right angles...
  • Communications Technology

    Communications Technology

    BASIC EQUIPMENT ACCT-BVP1-4. Students will be able to demonstrate proper set-up and use of basic production equipment. Demonstrate steps necessary to set-up, turn on, and operate equipment according to instructor's directions.
  • City of Virginia Beach: Turf and Water Quality

    City of Virginia Beach: Turf and Water Quality

    Virginia operates under a MS5 permit which limits fertilizer usage to municipalities and golf courses. Limit the use of Phosphorus used - if soil test call for 0% we use none for the year - in 2010 only 1% P...
  • Influencing Change through Collaboration Jerry D. VanVactor, DHA,

    Influencing Change through Collaboration Jerry D. VanVactor, DHA,

    Equalitarianism is a concept defined by Dr. Laurence J. Peter (you may have heard of the Peter Principle which states that everyone will promoted to his/her own level of incompetence); occupational incompetence is everywhere (SHARE HIS STORY OF SENDING HIS...
  • T Cell Development - The University of Vermont

    T Cell Development - The University of Vermont

    Word/Terms List ADCC Effector cells GVH reaction Perforin T Cell-mediated Immunity Principal function-Response to intracellular pathogens and cells expressing foreign antigens Recirculation-Naïve T cells circulate between the blood stream and the lymphatic system Antigen presentation-Naïve T cell cells only respond...
  • Tutorial Session - vulms.vu.edu.pk

    Tutorial Session - vulms.vu.edu.pk

    An English word can be divided into three parts: a prefix, a stem and a suffix. Prefix is what comes before the stem. A suffix is what is attached to the end of the stem. A stem is the root...
  • The Waste Land - جامعة الملك سعود

    The Waste Land - جامعة الملك سعود

    The Waste Land Literary Style. ModernismThe most important aspect of the work, and the one that informs all others, is the literary movement to which it belongs, modernism, which this work helped define. Modernism is the broad term used to...
  • An Exploration of Intuition Among Senior Leaders

    An Exploration of Intuition Among Senior Leaders

    Germinal Authors Perspective. In 1991, Behling and Eckel lamented that making sense of intuition as a management practice was hampered by its incongruities, despite its long consideration as an adjunct to leadership and decision making, those who wrote about it...