TDLT

From XiphWiki
Revision as of 17:06, 22 April 2013 by Gmaxwell (talk | contribs) (add some search results)
Jump to navigation Jump to search

This page holds the results of Time Domain Lapped Transform (TDLT) optimization problems looking for integer transform coefficients that provide optimal coding gain. Wherever possible the assumptions are stated. Later we should include testing against actual image data to verify the results (see test data here).

The coding gain objective used as the objective is taken from slide 13 of Tim's presentation An Introduction to Video Coding

<need figure with block matrix diagrams>

The free parameters are initially just the coefficients p_0,...,p_m,q_0,...,q_m where m=(n/2)-1. We limit these to being dyadic rationals, e.g., x/2^d with d=6, between [-1,1].

Given p's and q's and assuming a linear ramp constrains the s's.

4x8

Optimal real-valued coefficients for V:

p0 = -0.18117338915051454

q0 = 0.6331818230771687

CG = 8.60603

p0 q0 s0 s1 CG SBA Filterbank
R=f
6-bit
-11/64
-0.171875
36/64
0.5625
91/64
1.421875
85/64
1.328125
 
8.63473
 
22.0331
R=f
5-bit
-5/32
-0.15625
18/32
0.5625
46/32
1.4375
42/32
1.3125
 
8.63409
 
22.5715
File:4x8 5bit.png
R=t,D=f -12/64
-0.1875
41/64
0.640625
92/64
1.4375
1093/768
1.423177
 
8.60486
 
20.0573
R=t,D=t
8-bit
-32/256
-0.125
162/256
0.6328125
376/256
1.46875
357/256
1.39453125
 
8.60104
 
21.4037
File:4x8rd 8bit.png
R=t,D=t
7-bit
-32/128
-0.25
82/128
0.640625
184/128
1.4375
186/128
1.453125
 
8.59886
 
18.9411
File:4x8rd 7bit.png
R=t,D=t
6-bit
-16/64
-0.25
41/64
0.640625
92/64
1.4375
93/64
1.453125
 
8.59886
 
18.9411
R=t,D=t
5-bit
-8/32
-0.25
19/32
0.59375
52/32
1.625
47/32
1.46875
 
8.56068
 
20.3279
File:4x8rd 5bit.png
R=t,D=t
max SBA
-8/64
-0.125
30/64
0.46875
136/64
2.125
91/64
1.421875
 
8.23230
 
25.1934
File:4x8rd sba.png

8x16

Optimal real-valued coefficients for V:

p0 = -0.39460731547057293

p1 = -0.33002212811740816

p2 = -0.12391270981321137

q0 = 0.822154737511288

q1 = 0.632488694485779

q2 = 0.40214668677553894

CG = 9.56867

p0 p1 p2 q0 q1 q2 s0 s1 s2 s3 CG Filterbank
R=f
6-bit
-23/64
-0.359375
-18/64
-0.28125
-6/64
-0.09375
48/64
0.75
34/64
0.53125
20/64
0.3125
90/64
1.40625
73/64
1.140625
72/64
1.125
75/64
1.171875
 
9.60021
R=f
5-bit
-12/32
-0.375
-9/32
-0.28125
-4/32
-0.125
24/32
0.75
17/32
0.53125
10/32
45/32
1.40625
37/32
1.15625
36/32
38/32
1.1875
 
9.59946
File:8x16 5bit.png
R=t,D=f -26/64
-0.40625
-22/64
-0.34375
-8/64
-0.125
53/64
0.828125
41/64
0.640625
26/64
0.40625
11/8
1.375
879/768
1.14453125
1469/1280
1.14765625
275/224
1.2276785714285714
 
9.56627
R=t,D=t
8-bit
-96/256
-0.375
-83/256
-0.32421875
-36/256
-0.140625
210/256
0.8203125
160/256
0.625
104/256
0.40625
368/256
1.4375
302/256
1.1796875
293/256
1.14453125
317/256
1.23828125
 
9.56761
File:8x16rd 8bit.png
R=t,D=t
7-bit
-48/128
-0.375
-45/128
-0.3515625
-16/128
-0.125
105/128
0.8203125
80/128
0.625
53/128
0.4140625
184/128
1.4375
151/128
1.1796875
147/128
1.1484375
157/128
1.2265625
 
9.56672
File:8x16rd 7bit.png
R=t,D=t
6-bit
-24/64
-0.375
-20/64
-0.3125
-4/64
-0.0625
53/64
0.828125
40/64
0.625
24/64
0.375
88/64
1.375
75/64
1.171875
76/64
1.1875
76/64
1.1875
 
9.56161
R=t,D=t
5-bit
-12/32
-0.375
-10/32
-0.3125
-2/32
-0.0625
26/32
0.8125
20/32
0.625
12/32
0.375
48/32
1.5
38/32
1.1875
38/32
1.1875
38/32
1.1875
 
9.5596
File:8x16rd 5bit.png

16x32

Best-known real-valued coefficients for V (R=t):

p0 = -0.42111473798940136

p1 = -0.4121736499899753

p2 = -0.3350240707669929

p3 = -0.3224547931861314

p4 = -0.25883387978005545

p5 = -0.20951913473498104

p6 = -0.0598657149803332

q0 = 0.9107782439906195

q1 = 0.8109855829278226

q2 = 0.715846584586721

q3 = 0.6135951570714172

q4 = 0.49846644853347627

q5 = 0.3945215834922529

q6 = 0.21822275136248082

CG = 9.81157

p0 p1 p2 p3 p4 p5 p6 q0 q1 q2 q3 q4 q5 q6 s0 s1 s2 s3 s4 s5 s6 s7 CG Filterbank
R=f
6-bit
-24/64
-0.375
-23/64
-0.359375
-17/64
-0.265625
-12/64
-0.1875
-14/64
-0.21875
-13/64
-0.203125
-7/64
-0.109375
50/64
0.78125
40/64
0.625
31/64
0.484375
22/64
0.34375
18/64
0.28125
16/64
0.25
11/64
0.171875
90/64
1.40625
74/64
1.15625
73/64
1.140625
71/64
1.109375
67/64
1.046875
67/64
1.046875
67/64
1.046875
72/64
1.125
 
9.89338
R=t,D=f -26/64
-0.40625
-27/64
-0.421875
-22/64
-0.34375
-18/64
-0.28125
-16/64
-0.25
-14/64
-0.21875
-5/64
-0.078125
58/64
0.90625
52/64
0.8125
45/64
0.703125
36/64
0.5625
31/64
0.484375
23/64
0.359375
13/64
0.203125
3/2
1.5
77/64
1.203125
77/64
1.203125
1105/896
1.23326
218/192
1.135417
197/176
1.119318
1919/1664
1.153245
4351/3840
1.133073
 
9.79398
R=t,D=t
6-bit
-32/64
-0.5
-28/64
-0.4375
-24/64
-0.375
-32/64
-0.5
-24/64
-0.375
-13/64
-0.203125
-2/64
-0.03125
59/64
0.921875
53/64
0.828125
46/64
0.71875
41/64
0.640625
35/64
0.546875
24/64
0.375
12/64
0.1875
80/64
1.25
72/64
1.125
73/64
1.140625
68/64
1.0625
72/64
1.125
74/64
1.15625
74/64
1.15625
70/64
1.09375
 
9.78294

Type-IV Coding Gain

4x8 4x8 Ramp 8x16 8x16 Ramp 16x32 16x32 Ramp
Real Valued 8.6349 8.60603 9.6005 9.56867 9.9057 9.81157
Dyadic (8-bit) 8.60104 9.56761
Loss 0.00499 0.00105
Dyadic (7-bit) 8.59886 9.56672
Loss 0.00717 0.00195
Dyadic (6-bit) 8.63473 8.59886 9.60021 9.56161 9.89338 9.78294
Loss 0.00017 0.00717 0.00029 0.00706 0.01232 0.02863
Dyadic (5-bit) 8.63409 8.56068 9.59946 9.5596
Loss 0.00081 0.04535 0.00104 0.00907

8x16 Type-III

p0 p1 p2 q0 q1 q2 s0 s1 s2 s3 CG Filterbank
R=f
6-bit
-25/64
-0.390625
-20/64
-0.3125
-7/64
-0.109375
49/64
0.765625
35/64
0.546875
21/64
0.328125
90/64
1.40625
72/64
1.125
73/64
1.140625
76/64
1.1875
 
9.6112
R=f
5-bit
-13/32
-0.40625
-11/32
-0.34375
-4/32
-0.125
25/32
0.78125
18/32
0.5625
11/32
0.34375
45/32
1.40625
36/32
1.125
36/32
1.125
38/32
1.1875
 
9.61048
File:8x16 type3 5bit.png

16x32 Type-III

p0 p1 p2 p3 p4 p5 p6 q0 q1 q2 q3 q4 q5 q6 s0 s1 s2 s3 s4 s5 s6 s7 CG Filterbank
R=f
6-bit
-30/64
-0.46875
-35/64
-0.546875
-31/64
-0.484375
-29/64
-0.453125
-25/64
-0.390625
-19/64
-0.296875
-10/64
-0.15625
54/64
0.84375
45/64
0.703125
40/64
0.625
36/64
0.5625
32/64
0.5
25/64
0.390625
17/64
0.265625
90/64
1.40625
70/64
1.09375
69/64
1.078125
67/64
1.046875
67/64
1.046875
67/64
1.046875
68/64
1.0625
74/64
1.15625
 
9.94127
File:16x32 type3 6bit.png
R=f
5-bit
-15/32
-0.46875
-17/32
-0.53125
-14/32
-0.4375
-12/32
-0.375
-9/32
-0.28125
-5/32
-0.15625
0/32
0.0
27/32
0.84375
23/32
0.71875
20/32
0.625
17/32
0.53125
14/32
0.4375
9/32
0.28125
3/32
0.09375
45/32
1.40625
35/32
1.09375
35/32
1.09375
34/32
1.0625
34/32
1.0625
35/32
1.09375
36/32
1.125
33/32
1.03125
 
9.93998
File:16x32 type3 5bit.png

Type-III Coding Gain

4x8 8x16 16x32
Real Valued 8.6349 9.6115 9.9496
Dyadic (6-bit) 8.63473 9.6112 9.94127
Loss 0.00017 0.00030 0.00833
Dyadic (5-bit) 8.63409 9.61048 9.93998
Loss 0.00081 0.00102 0.00962

Simplex search results

Type-IV

  • 8x16 2D AR95
    • 19.199865874793673 { 89, 73, 72, 75,-23,-18, -6, 48, 34, 20}
  • 8x16 Subset 1
    • 13.9812271938690706 { 84, 68, 67, 68,-24,-19, -8, 38, 24, 13}
  • 8x16 Subset 3
    • 16.7631044694739657 { 85, 68, 67, 69,-24,-18, -9, 38, 24, 13}

Type-III =

  • 8x16 2D AR95
    • 19.2223200050370124 { 90, 72, 73, 76,-25,-20, -7, 49, 35, 21}
  • 8x16 Subset 1
    • 14.0121200303196343 { 86, 66, 67, 69,-28,-25,-11, 44, 28, 16}
  • 8x16 Subset 3
    • 16.8035257369844686 { 87, 66, 67, 70,-29,-24,-11, 44, 28, 15}