TDLT

This page holds the results of Time Domain Lapped Transform (TDLT) optimization problems looking for integer transform coefficients that provide optimal coding gain. Wherever possible the assumptions are stated. Later we should include testing against actual image data to verify the results (see test data here).

The coding gain objective used as the objective is taken from slide 13 of Tim's presentation An Introduction to Video Coding

<need figure with block matrix diagrams>

The free parameters are initially just the coefficients p_0,...,p_m,q_0,...,q_m where m=(n/2)-1. We limit these to being dyadic rationals, e.g., x/2^d with d=6, between [-1,1].

Given p's and q's and assuming a linear ramp constrains the s's.

4x8

Optimal real-valued coefficients for V:

p0 = -0.18117338915051454

q0 = 0.6331818230771687

CG = 8.60603

	p0	q0	s0	s1	CG	SBA	Filterbank
R=f 6-bit	-11/64 -0.171875	36/64 0.5625	91/64 1.421875	85/64 1.328125	8.63473	22.0331
R=f 5-bit	-5/32 -0.15625	18/32 0.5625	46/32 1.4375	42/32 1.3125	8.63409	22.5715	File:4x8 5bit.png
R=t,D=f	-12/64 -0.1875	41/64 0.640625	92/64 1.4375	1093/768 1.423177	8.60486	20.0573
R=t,D=t 8-bit	-32/256 -0.125	162/256 0.6328125	376/256 1.46875	357/256 1.39453125	8.60104	21.4037	File:4x8rd 8bit.png
R=t,D=t 7-bit	-32/128 -0.25	82/128 0.640625	184/128 1.4375	186/128 1.453125	8.59886	18.9411	File:4x8rd 7bit.png
R=t,D=t 6-bit	-16/64 -0.25	41/64 0.640625	92/64 1.4375	93/64 1.453125	8.59886	18.9411
R=t,D=t 5-bit	-8/32 -0.25	19/32 0.59375	52/32 1.625	47/32 1.46875	8.56068	20.3279	File:4x8rd 5bit.png
R=t,D=t max SBA	-8/64 -0.125	30/64 0.46875	136/64 2.125	91/64 1.421875	8.23230	25.1934	File:4x8rd sba.png

8x16

Optimal real-valued coefficients for V:

p0 = -0.39460731547057293

p1 = -0.33002212811740816

p2 = -0.12391270981321137

q0 = 0.822154737511288

q1 = 0.632488694485779

q2 = 0.40214668677553894

CG = 9.56867

	p0	p1	p2	q0	q1	q2	s0	s1	s2	s3	CG	Filterbank
R=f 6-bit	-23/64 -0.359375	-18/64 -0.28125	-6/64 -0.09375	48/64 0.75	34/64 0.53125	20/64 0.3125	90/64 1.40625	73/64 1.140625	72/64 1.125	75/64 1.171875	9.60021
R=f 5-bit	-12/32 -0.375	-9/32 -0.28125	-4/32 -0.125	24/32 0.75	17/32 0.53125	10/32	45/32 1.40625	37/32 1.15625	36/32	38/32 1.1875	9.59946	File:8x16 5bit.png
R=t,D=f	-26/64 -0.40625	-22/64 -0.34375	-8/64 -0.125	53/64 0.828125	41/64 0.640625	26/64 0.40625	11/8 1.375	879/768 1.14453125	1469/1280 1.14765625	275/224 1.2276785714285714	9.56627
R=t,D=t 7-bit	-48/128 -0.375	-45/128 -0.3515625	-16/128 -0.125	105/128 0.8203125	80/128 0.625	53/128 0.4140625	184/128 1.4375	151/128 1.1796875	147/128 1.1484375	157/128 1.2265625	9.56672	File:8x16rd 7bit.png
R=t,D=t 6-bit	-24/64 -0.375	-20/64 -0.3125	-4/64 -0.0625	53/64 0.828125	40/64 0.625	24/64 0.375	88/64 1.375	75/64 1.171875	76/64 1.1875	76/64 1.1875	9.56161
R=t,D=t 5-bit	-12/32 -0.375	-10/32 -0.3125	-2/32 -0.0625	26/32 0.8125	20/32 0.625	12/32 0.375	48/32 1.5	38/32 1.1875	38/32 1.1875	38/32 1.1875	9.5596	File:8x16rd 5bit.png

16x32

Best-known real-valued coefficients for V (R=t):

p0 = -0.42111473798940136

p1 = -0.4121736499899753

p2 = -0.3350240707669929

p3 = -0.3224547931861314

p4 = -0.25883387978005545

p5 = -0.20951913473498104

p6 = -0.0598657149803332

q0 = 0.9107782439906195

q1 = 0.8109855829278226

q2 = 0.715846584586721

q3 = 0.6135951570714172

q4 = 0.49846644853347627

q5 = 0.3945215834922529

q6 = 0.21822275136248082

CG = 9.81157

	p0	p1	p2	p3	p4	p5	p6	q0	q1	q2	q3	q4	q5	q6	s0	s1	s2	s3	s4	s5	s6	s7	CG	Filterbank
R=f 6-bit	-24/64 -0.375	-23/64 -0.359375	-17/64 -0.265625	-12/64 -0.1875	-14/64 -0.21875	-13/64 -0.203125	-7/64 -0.109375	50/64 0.78125	40/64 0.625	31/64 0.484375	22/64 0.34375	18/64 0.28125	16/64 0.25	11/64 0.171875	90/64 1.40625	74/64 1.15625	73/64 1.140625	71/64 1.109375	67/64 1.046875	67/64 1.046875	67/64 1.046875	72/64 1.125	9.89338
R=t,D=f	-26/64 -0.40625	-27/64 -0.421875	-22/64 -0.34375	-18/64 -0.28125	-16/64 -0.25	-14/64 -0.21875	-5/64 -0.078125	58/64 0.90625	52/64 0.8125	45/64 0.703125	36/64 0.5625	31/64 0.484375	23/64 0.359375	13/64 0.203125	3/2 1.5	77/64 1.203125	77/64 1.203125	1105/896 1.23326	218/192 1.135417	197/176 1.119318	1919/1664 1.153245	4351/3840 1.133073	9.79398
R=t,D=t 6-bit	-32/64 -0.5	-28/64 -0.4375	-24/64 -0.375	-32/64 -0.5	-24/64 -0.375	-13/64 -0.203125	-2/64 -0.03125	59/64 0.921875	53/64 0.828125	46/64 0.71875	41/64 0.640625	35/64 0.546875	24/64 0.375	12/64 0.1875	80/64 1.25	72/64 1.125	73/64 1.140625	68/64 1.0625	72/64 1.125	74/64 1.15625	74/64 1.15625	70/64 1.09375	9.78294

Type-IV Coding Gain

	4x8	4x8 Ramp	8x16	8x16 Ramp	16x32	16x32 Ramp
Real Valued	8.6349	8.60603	9.6005	9.56867	9.9057	9.81157
Dyadic (8-bit)		8.60104
Loss		0.00499
Dyadic (7-bit)		8.59886		9.56672
Loss		0.00717		0.00195
Dyadic (6-bit)	8.63473	8.59886	9.60021	9.56161	9.89338	9.78294
Loss	0.00017	0.00717	0.00029	0.00706	0.01232	0.02863
Dyadic (5-bit)	8.63409	8.56068	9.59946	9.5596
Loss	0.00081	0.04535	0.00104	0.00907

8x16 Type-III

	p0	p1	p2	q0	q1	q2	s0	s1	s2	s3	CG	Filterbank
R=f 6-bit	-25/64 -0.390625	-20/64 -0.3125	-7/64 -0.109375	49/64 0.765625	35/64 0.546875	21/64 0.328125	90/64 1.40625	72/64 1.125	73/64 1.140625	76/64 1.1875	9.6112
R=f 5-bit	-13/32 -0.40625	-11/32 -0.34375	-4/32 -0.125	25/32 0.78125	18/32 0.5625	11/32 0.34375	45/32 1.40625	36/32 1.125	36/32 1.125	38/32 1.1875	9.61048	File:8x16 type3 5bit.png

16x32 Type-III

	p0	p1	p2	p3	p4	p5	p6	q0	q1	q2	q3	q4	q5	q6	s0	s1	s2	s3	s4	s5	s6	s7	CG	Filterbank
R=f 6-bit	-30/64 -0.46875	-35/64 -0.546875	-31/64 -0.484375	-29/64 -0.453125	-25/64 -0.390625	-19/64 -0.296875	-10/64 -0.15625	54/64 0.84375	45/64 0.703125	40/64 0.625	36/64 0.5625	32/64 0.5	25/64 0.390625	17/64 0.265625	90/64 1.40625	70/64 1.09375	69/64 1.078125	67/64 1.046875	67/64 1.046875	67/64 1.046875	68/64 1.0625	74/64 1.15625	9.94127	File:16x32 type3 6bit.png
R=f 5-bit	-15/32 -0.46875	-17/32 -0.53125	-14/32 -0.4375	-12/32 -0.375	-9/32 -0.28125	-5/32 -0.15625	0/32 0.0	27/32 0.84375	23/32 0.71875	20/32 0.625	17/32 0.53125	14/32 0.4375	9/32 0.28125	3/32 0.09375	45/32 1.40625	35/32 1.09375	35/32 1.09375	34/32 1.0625	34/32 1.0625	35/32 1.09375	36/32 1.125	33/32 1.03125	9.93998	File:16x32 type3 5bit.png

Type-III Coding Gain

	4x8	8x16	16x32
Real Valued	8.6349	9.6115	9.9496
Dyadic (6-bit)	8.63473	9.6112	9.94127
Loss	0.00017	0.00030	0.00833
Dyadic (5-bit)	8.63409	9.61048	9.93998
Loss	0.00081	0.00102	0.00962