TDLT: Difference between revisions

From XiphWiki
Jump to navigation Jump to search
m (better name for this)
No edit summary
 
(34 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This page holds the results of Time Domain Lapped Transform (TDLT) optimization problems looking for integer transform coefficients that provide optimal coding gain.  Wherever possible the assumptions are stated.  Later we should include testing against actual image data to verify the results (see test data [http://people.xiph.org/~tterribe/tmp/subset1-y4m.tar.gz here]).
This page holds the results of Time Domain Lapped Transform (TDLT) optimization problems looking for integer transform coefficients that provide optimal coding gain.  All numbers are generated against AR95 unless otherwise stated. Wherever possible the assumptions are stated.  Later we should include testing against actual image data to verify the results (see test data [http://people.xiph.org/~tterribe/tmp/subset1-y4m.tar.gz here]).


The coding gain objective used as the objective is taken from slide 13 of Tim's presentation [http://people.xiph.org/~tterribe/pubs/lca2012/auckland/intro_to_video1.pdf An Introduction to Video Coding]
The coding gain objective used as the objective is taken from slide 13 of Tim's presentation [http://people.xiph.org/~tterribe/pubs/lca2012/auckland/intro_to_video1.pdf An Introduction to Video Coding]
Line 26: Line 26:
!s1
!s1
!CG
!CG
!SBA
!Filterbank
|-
|-
|R=f
|R=f<br>6-bit
| -11/64<br>-0.171875
| -11/64<br>-0.171875
| 36/64<br>0.5625
| 36/64<br>0.5625
Line 33: Line 35:
| 85/64<br>1.328125
| 85/64<br>1.328125
| &nbsp;<br>8.63473
| &nbsp;<br>8.63473
| &nbsp;<br>22.0331
| [[Image:4x8.png|64px]]
|-
|R=f<br>5-bit
| -5/32<br>-0.15625
| 18/32<br>0.5625
| 46/32<br>1.4375
| 42/32<br>1.3125
| &nbsp;<br>8.63409
| &nbsp;<br>22.5715
| [[Image:4x8_5bit.png|64px]]
|-
|-
|R=t,D=f
|R=t,D=f
Line 40: Line 53:
| 1093/768<br>1.423177
| 1093/768<br>1.423177
| &nbsp;<br>8.60486
| &nbsp;<br>8.60486
| &nbsp;<br>20.0573
| [[Image:4x8r.png|64px]]
|-
|R=t,D=t<br>8-bit
| -32/256<br>-0.125
| 162/256<br>0.6328125
| 376/256<br>1.46875
| 357/256<br>1.39453125
| &nbsp;<br>8.60104
| &nbsp;<br>21.4037
| [[Image:4x8rd_8bit.png|64px]]
|-
|R=t,D=t<br>7-bit
| -32/128<br>-0.25
| 82/128<br>0.640625
| 184/128<br>1.4375
| 186/128<br>1.453125
| &nbsp;<br>8.59886
| &nbsp;<br>18.9411
| [[Image:4x8rd_7bit.png|64px]]
|-
|-
|R=t,D=t
|R=t,D=t<br>6-bit
| -16/64<br>-0.25
| -16/64<br>-0.25
| 41/64<br>0.640625
| 41/64<br>0.640625
Line 47: Line 80:
| 93/64<br>1.453125
| 93/64<br>1.453125
| &nbsp;<br>8.59886
| &nbsp;<br>8.59886
| &nbsp;<br>18.9411
| [[Image:4x8rd.png|64px]]
|-
|R=t,D=t<br>5-bit
| -8/32<br>-0.25
| 19/32<br>0.59375
| 52/32<br>1.625
| 47/32<br>1.46875
| &nbsp;<br>8.56068
| &nbsp;<br>20.3279
| [[Image:4x8rd_5bit.png|64px]]
|-
|R=t,D=t<br>max SBA
| -8/64<br>-0.125
| 30/64<br>0.46875
| 136/64<br>2.125
| 91/64<br>1.421875
| &nbsp;<br>8.23230
| &nbsp;<br>25.1934
| [[Image:4x8rd_sba.png|64px]]
|}
|}


Line 82: Line 135:
!Filterbank
!Filterbank
|-
|-
|R=f
|R=f<br>6-bit
| -23/64<br>-0.359375
| -23/64<br>-0.359375
| -18/64<br>-0.28125
| -18/64<br>-0.28125
Line 95: Line 148:
| &nbsp;<br>9.60021
| &nbsp;<br>9.60021
| [[Image:8x16.png|64px]]
| [[Image:8x16.png|64px]]
|-
|R=f<br>5-bit
| -12/32<br>-0.375
| -9/32<br>-0.28125
| -4/32<br>-0.125
| 24/32<br>0.75
| 17/32<br>0.53125
| 10/32<br>
| 45/32<br>1.40625
| 37/32<br>1.15625
| 36/32<br>
| 38/32<br>1.1875
| &nbsp;<br>9.59946
| [[Image:8x16_5bit.png|64px]]
|-
|-
|R=t,D=f
|R=t,D=f
Line 110: Line 177:
| [[Image:8x16r.png|64px]]
| [[Image:8x16r.png|64px]]
|-
|-
|R=t,D=t
|R=t,D=t<br>8-bit
| -96/256<br>-0.375
| -83/256<br>-0.32421875
| -36/256<br>-0.140625
| 210/256<br>0.8203125
| 160/256<br>0.625
| 104/256<br>0.40625
| 368/256<br>1.4375
| 302/256<br>1.1796875
| 293/256<br>1.14453125
| 317/256<br>1.23828125
| &nbsp;<br>9.56761
| [[Image:8x16rd_8bit.png|64px]]
|-
|R=t,D=t<br>7-bit
| -48/128<br>-0.375
| -45/128<br>-0.3515625
| -16/128<br>-0.125
| 105/128<br>0.8203125
| 80/128<br>0.625
| 53/128<br>0.4140625
| 184/128<br>1.4375
| 151/128<br>1.1796875
| 147/128<br>1.1484375
| 157/128<br>1.2265625
| &nbsp;<br>9.56672
| [[Image:8x16rd_7bit.png|64px]]
|-
|R=t,D=t<br>6-bit
| -24/64<br>-0.375
| -24/64<br>-0.375
| -20/64<br>-0.3125
| -20/64<br>-0.3125
Line 123: Line 218:
| &nbsp;<br>9.56161
| &nbsp;<br>9.56161
| [[Image:8x16rd.png|64px]]
| [[Image:8x16rd.png|64px]]
|-
|R=t,D=t<br>5-bit
| -12/32<br>-0.375
| -10/32<br>-0.3125
| -2/32<br>-0.0625
| 26/32<br>0.8125
| 20/32<br>0.625
| 12/32<br>0.375
| 48/32<br>1.5
| 38/32<br>1.1875
| 38/32<br>1.1875
| 38/32<br>1.1875
| &nbsp;<br>9.5596
| [[Image:8x16rd_5bit.png|64px]]
|}
|}


Line 184: Line 293:
!s7
!s7
!CG
!CG
!Filterbank
|-
|-
|R=f
|R=f<br>6-bit
| -24/64<br>-0.375
| -23/64<br>-0.359375
| -17/64<br>-0.265625
| -12/64<br>-0.1875
| -14/64<br>-0.21875
| -13/64<br>-0.203125
| -7/64<br>-0.109375
| 50/64<br>0.78125
| 40/64<br>0.625
| 31/64<br>0.484375
| 22/64<br>0.34375
| 18/64<br>0.28125
| 16/64<br>0.25
| 11/64<br>0.171875
| 90/64<br>1.40625
| 74/64<br>1.15625
| 73/64<br>1.140625
| 71/64<br>1.109375
| 67/64<br>1.046875
| 67/64<br>1.046875
| 67/64<br>1.046875
| 72/64<br>1.125
| &nbsp;<br>9.89338
| [[Image:16x32.png|64px]]
|-
|-
|R=t,D=f
|R=t,D=f
| -26/64<br>-0.40625
| -26/64<br>-0.40625
| -23/64<br>-0.359375
| -27/64<br>-0.421875
| -20/64<br>-0.3125
| -22/64<br>-0.34375
| -18/64<br>-0.28125
| -18/64<br>-0.28125
| -16/64<br>-0.25
| -14/64<br>-0.21875
| -14/64<br>-0.21875
| -14/64<br>-0.21875
| -5/64<br>-0.078125
| -2/64<br>-0.03125
| 58/64<br>0.90625
| 58/64<br>0.90625
| 52/64<br>0.8125
| 52/64<br>0.8125
Line 201: Line 335:
| 31/64<br>0.484375
| 31/64<br>0.484375
| 23/64<br>0.359375
| 23/64<br>0.359375
| 16/64<br>0.25
| 13/64<br>0.203125
| 3/2<br>1.5
| 3/2<br>1.5
| 77/64<br>1.20313
| 77/64<br>1.203125
| 373/320<br>1.16563
| 77/64<br>1.203125
| 543/448<br>1.21205
| 1105/896<br>1.23326
| 109/96<br>1.13542
| 218/192<br>1.135417
| 1543/1408<br>1.09588
| 197/176<br>1.119318
| 1823/1664<br>1.09555
| 1919/1664<br>1.153245
| 131/120<br>1.09167
| 4351/3840<br>1.133073
| &nbsp;<br>9.79008
| &nbsp;<br>9.79398
| [[Image:16x32r.png|64px]]
|-
|R=t,D=t<br>6-bit
| -32/64<br>-0.5
| -28/64<br>-0.4375
| -24/64<br>-0.375
| -32/64<br>-0.5
| -24/64<br>-0.375
| -13/64<br>-0.203125
| -2/64<br>-0.03125
| 59/64<br>0.921875
| 53/64<br>0.828125
| 46/64<br>0.71875
| 41/64<br>0.640625
| 35/64<br>0.546875
| 24/64<br>0.375
| 12/64<br>0.1875
| 80/64<br>1.25
| 72/64<br>1.125
| 73/64<br>1.140625
| 68/64<br>1.0625
| 72/64<br>1.125
| 74/64<br>1.15625
| 74/64<br>1.15625
| 70/64<br>1.09375
| &nbsp;<br>9.78294
| [[Image:16x32rd.png|64px]]
|}
 
== Type-IV Coding Gain ==
 
{|
!
!4x8
!4x8 Ramp
!8x16
!8x16 Ramp
!16x32
!16x32 Ramp
|-
|Real Valued
|8.6349
|8.60603
|9.6005
|9.56867
|9.9057
|9.81157
|-
|Dyadic (8-bit)
|
|8.60104
|
|9.56761
|
|
|-
|Loss
|
|0.00499
|
|0.00105
|
|
|-
|Dyadic (7-bit)
|
|8.59886
|
|9.56672
|
|
|-
|Loss
|
|0.00717
|
|0.00195
|
|
|-
|Dyadic (6-bit)
|8.63473
|8.59886
|9.60021
|9.56161
|9.89338
|9.78294
|-
|Loss
|0.00017
|0.00717
|0.00029
|0.00706
|0.01232
|0.02863
|-
|Dyadic (5-bit)
|8.63409
|8.56068
|9.59946
|9.5596
|
|
|-
|-
|R=t,D=t
|Loss
|0.00081
|0.04535
|0.00104
|0.00907
|
|
|}
|}
== 8x16 Type-III ==
{|
!
!p0
!p1
!p2
!q0
!q1
!q2
!s0
!s1
!s2
!s3
!CG
!Filterbank
|-
|R=f<br>6-bit
| -25/64<br>-0.390625
| -20/64<br>-0.3125
| -7/64<br>-0.109375
| 49/64<br>0.765625
| 35/64<br>0.546875
| 21/64<br>0.328125
| 90/64<br>1.40625
| 72/64<br>1.125
| 73/64<br>1.140625
| 76/64<br>1.1875
| &nbsp;<br>9.6112
| [[Image:8x16_type3.png|64px]]
|-
|R=f<br>5-bit
| -13/32<br>-0.40625
| -11/32<br>-0.34375
| -4/32<br>-0.125
| 25/32<br>0.78125
| 18/32<br>0.5625
| 11/32<br>0.34375
| 45/32<br>1.40625
| 36/32<br>1.125
| 36/32<br>1.125
| 38/32<br>1.1875
| &nbsp;<br>9.61048
| [[Image:8x16_type3_5bit.png|64px]]
|}
== 16x32 Type-III ==
{|
!
!p0
!p1
!p2
!p3
!p4
!p5
!p6
!q0
!q1
!q2
!q3
!q4
!q5
!q6
!s0
!s1
!s2
!s3
!s4
!s5
!s6
!s7
!CG
!Filterbank
|-
|R=f<br>6-bit
| -30/64<br>-0.46875
| -35/64<br>-0.546875
| -31/64<br>-0.484375
| -29/64<br>-0.453125
| -25/64<br>-0.390625
| -19/64<br>-0.296875
| -10/64<br>-0.15625
| 54/64<br>0.84375
| 45/64<br>0.703125
| 40/64<br>0.625
| 36/64<br>0.5625
| 32/64<br>0.5
| 25/64<br>0.390625
| 17/64<br>0.265625
| 90/64<br>1.40625
| 70/64<br>1.09375
| 69/64<br>1.078125
| 67/64<br>1.046875
| 67/64<br>1.046875
| 67/64<br>1.046875
| 68/64<br>1.0625
| 74/64<br>1.15625
| &nbsp;<br>9.94127
| [[Image:16x32_type3_6bit.png|64px]]
|-
|R=f<br>5-bit
| -15/32<br>-0.46875
| -17/32<br>-0.53125
| -14/32<br>-0.4375
| -12/32<br>-0.375
| -9/32<br>-0.28125
| -5/32<br>-0.15625
| 0/32<br>0.0
| 27/32<br>0.84375
| 23/32<br>0.71875
| 20/32<br>0.625
| 17/32<br>0.53125
| 14/32<br>0.4375
| 9/32<br>0.28125
| 3/32<br>0.09375
| 45/32<br>1.40625
| 35/32<br>1.09375
| 35/32<br>1.09375
| 34/32<br>1.0625
| 34/32<br>1.0625
| 35/32<br>1.09375
| 36/32<br>1.125
| 33/32<br>1.03125
| &nbsp;<br>9.93998
| [[Image:16x32_type3_5bit.png|64px]]
|}
== Type-III Coding Gain ==
{|
!
!4x8
!8x16
!16x32
|-
|Real Valued
|8.6349
|9.6115
|9.9496
|-
|Dyadic (6-bit)
|8.63473
|9.6112
|9.94127
|-
|Loss
|0.00017
|0.00030
|0.00833
|-
|Dyadic (5-bit)
|8.63409
|9.61048
|9.93998
|-
|Loss
|0.00081
|0.00102
|0.00962
|}
== Simplex search results ==
=== Type-IV ===
* 8x16 2D AR95
** 19.199865874793673  { 89, 73, 72, 75,-23,-18, -6, 48, 34, 20}
* 8x16 Subset 1
** 13.9812271938690706 { 84, 68, 67, 68,-24,-19, -8, 38, 24, 13}
* 8x16 Subset 3
** 16.7631044694739657 { 85, 68, 67, 69,-24,-18, -9, 38, 24, 13}
=== Type-III ===
* 8x16 2D AR95
** 19.2223200050370124 { 90, 72, 73, 76,-25,-20, -7, 49, 35, 21}
* 8x16 Subset 1
** 14.0121200303196343 { 86, 66, 67, 69,-28,-25,-11, 44, 28, 16}
* 8x16 Subset 3
** 16.8035257369844686 { 87, 66, 67, 70,-29,-24,-11, 44, 28, 15}
[[Category:Daala]]

Latest revision as of 15:20, 18 August 2015

This page holds the results of Time Domain Lapped Transform (TDLT) optimization problems looking for integer transform coefficients that provide optimal coding gain. All numbers are generated against AR95 unless otherwise stated. Wherever possible the assumptions are stated. Later we should include testing against actual image data to verify the results (see test data here).

The coding gain objective used as the objective is taken from slide 13 of Tim's presentation An Introduction to Video Coding

<need figure with block matrix diagrams>

The free parameters are initially just the coefficients p_0,...,p_m,q_0,...,q_m where m=(n/2)-1. We limit these to being dyadic rationals, e.g., x/2^d with d=6, between [-1,1].

Given p's and q's and assuming a linear ramp constrains the s's.

4x8

Optimal real-valued coefficients for V:

p0 = -0.18117338915051454

q0 = 0.6331818230771687

CG = 8.60603

p0 q0 s0 s1 CG SBA Filterbank
R=f
6-bit 		-11/64
-0.171875 		36/64
0.5625 		91/64
1.421875 		85/64
1.328125 		 
8.63473 		 
22.0331
R=f
5-bit 		-5/32
-0.15625 		18/32
0.5625 		46/32
1.4375 		42/32
1.3125 		 
8.63409 		 
22.5715 		File:4x8 5bit.png
R=t,D=f -12/64
-0.1875 		41/64
0.640625 		92/64
1.4375 		1093/768
1.423177 		 
8.60486 		 
20.0573
R=t,D=t
8-bit 		-32/256
-0.125 		162/256
0.6328125 		376/256
1.46875 		357/256
1.39453125 		 
8.60104 		 
21.4037 		File:4x8rd 8bit.png
R=t,D=t
7-bit 		-32/128
-0.25 		82/128
0.640625 		184/128
1.4375 		186/128
1.453125 		 
8.59886 		 
18.9411 		File:4x8rd 7bit.png
R=t,D=t
6-bit 		-16/64
-0.25 		41/64
0.640625 		92/64
1.4375 		93/64
1.453125 		 
8.59886 		 
18.9411
R=t,D=t
5-bit 		-8/32
-0.25 		19/32
0.59375 		52/32
1.625 		47/32
1.46875 		 
8.56068 		 
20.3279 		File:4x8rd 5bit.png
R=t,D=t
max SBA 		-8/64
-0.125 		30/64
0.46875 		136/64
2.125 		91/64
1.421875 		 
8.23230 		 
25.1934 		File:4x8rd sba.png

8x16

Optimal real-valued coefficients for V:

p0 = -0.39460731547057293

p1 = -0.33002212811740816

p2 = -0.12391270981321137

q0 = 0.822154737511288

q1 = 0.632488694485779

q2 = 0.40214668677553894

CG = 9.56867

p0 p1 p2 q0 q1 q2 s0 s1 s2 s3 CG Filterbank
R=f
6-bit 		-23/64
-0.359375 		-18/64
-0.28125 		-6/64
-0.09375 		48/64
0.75 		34/64
0.53125 		20/64
0.3125 		90/64
1.40625 		73/64
1.140625 		72/64
1.125 		75/64
1.171875 		 
9.60021
R=f
5-bit 		-12/32
-0.375 		-9/32
-0.28125 		-4/32
-0.125 		24/32
0.75 		17/32
0.53125 		10/32
 45/32
1.40625 		37/32
1.15625 		36/32
 38/32
1.1875 		 
9.59946 		File:8x16 5bit.png
R=t,D=f -26/64
-0.40625 		-22/64
-0.34375 		-8/64
-0.125 		53/64
0.828125 		41/64
0.640625 		26/64
0.40625 		11/8
1.375 		879/768
1.14453125 		1469/1280
1.14765625 		275/224
1.2276785714285714 		 
9.56627
R=t,D=t
8-bit 		-96/256
-0.375 		-83/256
-0.32421875 		-36/256
-0.140625 		210/256
0.8203125 		160/256
0.625 		104/256
0.40625 		368/256
1.4375 		302/256
1.1796875 		293/256
1.14453125 		317/256
1.23828125 		 
9.56761 		File:8x16rd 8bit.png
R=t,D=t
7-bit 		-48/128
-0.375 		-45/128
-0.3515625 		-16/128
-0.125 		105/128
0.8203125 		80/128
0.625 		53/128
0.4140625 		184/128
1.4375 		151/128
1.1796875 		147/128
1.1484375 		157/128
1.2265625 		 
9.56672 		File:8x16rd 7bit.png
R=t,D=t
6-bit 		-24/64
-0.375 		-20/64
-0.3125 		-4/64
-0.0625 		53/64
0.828125 		40/64
0.625 		24/64
0.375 		88/64
1.375 		75/64
1.171875 		76/64
1.1875 		76/64
1.1875 		 
9.56161
R=t,D=t
5-bit 		-12/32
-0.375 		-10/32
-0.3125 		-2/32
-0.0625 		26/32
0.8125 		20/32
0.625 		12/32
0.375 		48/32
1.5 		38/32
1.1875 		38/32
1.1875 		38/32
1.1875 		 
9.5596 		File:8x16rd 5bit.png

16x32

Best-known real-valued coefficients for V (R=t):

p0 = -0.42111473798940136

p1 = -0.4121736499899753

p2 = -0.3350240707669929

p3 = -0.3224547931861314

p4 = -0.25883387978005545

p5 = -0.20951913473498104

p6 = -0.0598657149803332

q0 = 0.9107782439906195

q1 = 0.8109855829278226

q2 = 0.715846584586721

q3 = 0.6135951570714172

q4 = 0.49846644853347627

q5 = 0.3945215834922529

q6 = 0.21822275136248082

CG = 9.81157

p0 p1 p2 p3 p4 p5 p6 q0 q1 q2 q3 q4 q5 q6 s0 s1 s2 s3 s4 s5 s6 s7 CG Filterbank
R=f
6-bit 		-24/64
-0.375 		-23/64
-0.359375 		-17/64
-0.265625 		-12/64
-0.1875 		-14/64
-0.21875 		-13/64
-0.203125 		-7/64
-0.109375 		50/64
0.78125 		40/64
0.625 		31/64
0.484375 		22/64
0.34375 		18/64
0.28125 		16/64
0.25 		11/64
0.171875 		90/64
1.40625 		74/64
1.15625 		73/64
1.140625 		71/64
1.109375 		67/64
1.046875 		67/64
1.046875 		67/64
1.046875 		72/64
1.125 		 
9.89338
R=t,D=f -26/64
-0.40625 		-27/64
-0.421875 		-22/64
-0.34375 		-18/64
-0.28125 		-16/64
-0.25 		-14/64
-0.21875 		-5/64
-0.078125 		58/64
0.90625 		52/64
0.8125 		45/64
0.703125 		36/64
0.5625 		31/64
0.484375 		23/64
0.359375 		13/64
0.203125 		3/2
1.5 		77/64
1.203125 		77/64
1.203125 		1105/896
1.23326 		218/192
1.135417 		197/176
1.119318 		1919/1664
1.153245 		4351/3840
1.133073 		 
9.79398
R=t,D=t
6-bit 		-32/64
-0.5 		-28/64
-0.4375 		-24/64
-0.375 		-32/64
-0.5 		-24/64
-0.375 		-13/64
-0.203125 		-2/64
-0.03125 		59/64
0.921875 		53/64
0.828125 		46/64
0.71875 		41/64
0.640625 		35/64
0.546875 		24/64
0.375 		12/64
0.1875 		80/64
1.25 		72/64
1.125 		73/64
1.140625 		68/64
1.0625 		72/64
1.125 		74/64
1.15625 		74/64
1.15625 		70/64
1.09375 		 
9.78294

Type-IV Coding Gain

4x8 4x8 Ramp 8x16 8x16 Ramp 16x32 16x32 Ramp
Real Valued 8.6349 8.60603 9.6005 9.56867 9.9057 9.81157
Dyadic (8-bit) 8.60104 9.56761
Loss 0.00499 0.00105
Dyadic (7-bit) 8.59886 9.56672
Loss 0.00717 0.00195
Dyadic (6-bit) 8.63473 8.59886 9.60021 9.56161 9.89338 9.78294
Loss 0.00017 0.00717 0.00029 0.00706 0.01232 0.02863
Dyadic (5-bit) 8.63409 8.56068 9.59946 9.5596
Loss 0.00081 0.04535 0.00104 0.00907

8x16 Type-III

p0 p1 p2 q0 q1 q2 s0 s1 s2 s3 CG Filterbank
R=f
6-bit 		-25/64
-0.390625 		-20/64
-0.3125 		-7/64
-0.109375 		49/64
0.765625 		35/64
0.546875 		21/64
0.328125 		90/64
1.40625 		72/64
1.125 		73/64
1.140625 		76/64
1.1875 		 
9.6112
R=f
5-bit 		-13/32
-0.40625 		-11/32
-0.34375 		-4/32
-0.125 		25/32
0.78125 		18/32
0.5625 		11/32
0.34375 		45/32
1.40625 		36/32
1.125 		36/32
1.125 		38/32
1.1875 		 
9.61048 		File:8x16 type3 5bit.png

16x32 Type-III

p0 p1 p2 p3 p4 p5 p6 q0 q1 q2 q3 q4 q5 q6 s0 s1 s2 s3 s4 s5 s6 s7 CG Filterbank
R=f
6-bit 		-30/64
-0.46875 		-35/64
-0.546875 		-31/64
-0.484375 		-29/64
-0.453125 		-25/64
-0.390625 		-19/64
-0.296875 		-10/64
-0.15625 		54/64
0.84375 		45/64
0.703125 		40/64
0.625 		36/64
0.5625 		32/64
0.5 		25/64
0.390625 		17/64
0.265625 		90/64
1.40625 		70/64
1.09375 		69/64
1.078125 		67/64
1.046875 		67/64
1.046875 		67/64
1.046875 		68/64
1.0625 		74/64
1.15625 		 
9.94127 		File:16x32 type3 6bit.png
R=f
5-bit 		-15/32
-0.46875 		-17/32
-0.53125 		-14/32
-0.4375 		-12/32
-0.375 		-9/32
-0.28125 		-5/32
-0.15625 		0/32
0.0 		27/32
0.84375 		23/32
0.71875 		20/32
0.625 		17/32
0.53125 		14/32
0.4375 		9/32
0.28125 		3/32
0.09375 		45/32
1.40625 		35/32
1.09375 		35/32
1.09375 		34/32
1.0625 		34/32
1.0625 		35/32
1.09375 		36/32
1.125 		33/32
1.03125 		 
9.93998 		File:16x32 type3 5bit.png

Type-III Coding Gain

4x8 8x16 16x32
Real Valued 8.6349 9.6115 9.9496
Dyadic (6-bit) 8.63473 9.6112 9.94127
Loss 0.00017 0.00030 0.00833
Dyadic (5-bit) 8.63409 9.61048 9.93998
Loss 0.00081 0.00102 0.00962

Simplex search results

Type-IV

  • 8x16 2D AR95
    • 19.199865874793673 { 89, 73, 72, 75,-23,-18, -6, 48, 34, 20}
  • 8x16 Subset 1
    • 13.9812271938690706 { 84, 68, 67, 68,-24,-19, -8, 38, 24, 13}
  • 8x16 Subset 3
    • 16.7631044694739657 { 85, 68, 67, 69,-24,-18, -9, 38, 24, 13}

Type-III

  • 8x16 2D AR95
    • 19.2223200050370124 { 90, 72, 73, 76,-25,-20, -7, 49, 35, 21}
  • 8x16 Subset 1
    • 14.0121200303196343 { 86, 66, 67, 69,-28,-25,-11, 44, 28, 16}
  • 8x16 Subset 3
    • 16.8035257369844686 { 87, 66, 67, 70,-29,-24,-11, 44, 28, 15}
Retrieved from "https://wiki.xiph.org/index.php?title=TDLT&oldid=15972"