TDLT: Difference between revisions

From XiphWiki
Jump to navigation Jump to search
No edit summary
Line 42: Line 42:
Optimal real-valued coefficients for V:
Optimal real-valued coefficients for V:


p0 = -0.3625885819608689
p0 = -0.4045289182698497


p1 = -0.41911703104506826
p1 = -0.33468137871117265


p2 = -0.06962862896922385
p2 = -0.12545831989246597


q0 = 0.8423686986993569
q0 = 0.8222196599456361


q1 = 0.6393969893363041
q1 = 0.634500829289973


q2 = 0.3894929113709107
q2 = 0.4034606713167927


CG = 9.52441
CG = 9.56856


Optimal [maybe] integer-valued coefficients (d=6) for V:
Optimal [maybe] integer-valued coefficients (d=6) for V:

Revision as of 14:51, 12 May 2012

This page holds the results of Time Domain Lapped Transform (TDLT) optimization problems looking for integer transform coefficients that provide optimal coding gain. Wherever possible the assumptions are stated. Later we should include testing against actual image data to verify the results (see test data here).

The coding gain objective used as the objective is taken from slide 13 of Tim's presentation An Introduction to Video Coding

<need figure with block matrix diagrams>

The free parameters are initially just the coefficients p_0,...,p_m,q_0,...,q_m where m=(n/2)-1. We limit these to being dyadic rationals, e.g., x/2^d with d=6, between [-1,1].

Given p's and q's and assuming a linear ramp constrains the s's.

4x8

s0 = 4*(1-q0) s1 = 4*(1-p0*(1-q0))/3

Optimal real-valued coefficients for V:

p0 = -0.18117338915051454

q0 = 0.6331818230771687

CG = 8.60603

Optimal integer-valued coefficients (d=6) for V:

p0 = -12/64 = -0.1875

q0 = 41/64 = 0.640625

CG = 8.60486

Optimal integer-valued coefficients (d=6) were (1-p0*(1-q0)) is divisible by 3:

p0 = -13/64 = -0.203125

q0 = 41/64 = 0.640625

CG = 8.60446

8x16

Optimal real-valued coefficients for V:

p0 = -0.4045289182698497

p1 = -0.33468137871117265

p2 = -0.12545831989246597

q0 = 0.8222196599456361

q1 = 0.634500829289973

q2 = 0.4034606713167927

CG = 9.56856

Optimal [maybe] integer-valued coefficients (d=6) for V:

p0 = -26/64 = -0.40625

p1 = -22/64 = -0.34375

p2 = -8/64 = -0.125

q0 = 53/64 = 0.828125

q1 = 41/64 = 0.640625

q2 = 26/64 = 0.40625

9.56627