关于javascript：为什么将gl.texImage2D中的格式设置为gl.LUMINANCE而不是gl.RGB，使得画布上的Blob在文件大小中仅缩小5％？

Why setting format in gl.texImage2D to gl.LUMINANCE instead of gl.RGB makes the blob made out of the canvas only ~5% smaller in filesize?

在浏览javascript时，我遇到了一个令人困惑的问题。序言是：我使用ImageBitmap接口将不同mime类型的图像(主要是png / jpg)转换为位图，然后将其传输给worker以在单独的线程中转换为blob(为此，我首先将其绘制到屏幕外的画布上下文中)，然后保存到IDB中，而主线程继续加载新图像。在这样做时，为了拓宽视野，我决定在画布中使用webgl2渲染上下文，因为GL是我从未接触过的东西。

要将位图应用于画布，我使用了texImage2D函数，我似乎不明白。在这里，我可以指定存储在内存中的数据格式，该格式将呈现给GLSL(由于位图是在不使用Alpha预先乘以的情况下创建的，因此应该为rgb(right？))，内部格式和类型。由于格式/内部格式/类型的组合是由规范指定的，因此我尝试利用它们的多种功能，并为我的目的选择了最佳的(质量/文件大小)。由于要转换为位图的图像大部分是黑白图像，因此我认为亮度是我所需要的。但是首先我使用标准的RGB格式：

1
2
3

gl.texImage2D(
gl.TEXTURE_2D, 0, gl.RGB, bitmap.width, bitmap.height, 0, gl.RGB, gl.UNSIGNED_BYTE, bitmap
);

然后，我将RGB565与UNSIGNED_SHORT_5_6_5数据类型一起使用，并且在将斑点大小从RGB减小约30％的同时，没有看到任何质量损失。据我了解，它减少了，因为RGB565是每个像素2个无符号短字节，对吗？然后，我使用了UNSIGNED_SHORT_5_5_5_1 RGBA，与标准RGB相比，blob文件大小减少了约43％。甚至比RGB565还小！但是图像上的渐变变得古怪，所以我没有5551RGBA。我不理解5551 RGBA和RGB565在尺寸上的巨大差异。更令人困惑的是，根据规格类型/格式/内部格式组合使用亮度时，与标准RGB相比仅减少了5％。为什么RGB565的尺寸会降低30％，而亮度只有5％？

为此，我在片段着色器中使用了相同的浮点采样器：

1
2
3
4
5
6
7
8
9
10
11
12

#version 300 es
precision mediump float;
precision mediump sampler2D;

uniform sampler2D sampler;
uniform vec2 dimensions;

out vec4 color;

void main(){
color = texture(sampler, vec2(gl_FragCoord.x/dimensions.x, 1.0 - (gl_FragCoord.y/dimensions.y)));
}

还有相同的pixelStorei和texParameteri：

1
2
3
4
5
6
7

gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, false);
gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false);
gl.pixelStorei(gl.UNPACK_COLORSPACE_CONVERSION_WEBGL, gl.NONE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);

如下图所示，如果图像是黑白的，亮度不会改变blob的文件大小，而如果是彩色的，则亮度会明显降低，尽管比RGBA4还要小。考虑到RGBA4每个像素有2个字节，而LUMA-1。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200

(async() => {
function createImage(src) {
\treturn new Promise((rs, rj) => {
\t\tvar img = new Image();
\t\timg.crossOrigin = 'anonymous';
\t\timg.src = src;
\t\timg.onload = () => rs(img);
\t\timg.onerror = e => rj(e);
});
};

var jpeg = await createImage('https://upload.wikimedia.org/wikipedia/commons/a/aa/5inchHowitzerFiringGallipoli1915.jpeg');

var png = await createImage('https://upload.wikimedia.org/wikipedia/commons/2/2c/6.d%C3%ADl_html_m2fdede78.png');

var jpgClr = await createImage('https://upload.wikimedia.org/wikipedia/commons/thumb/e/ed/%22Good_bye%2C_sweetheart%22%2C_tobacco_label%2C_ca._1865.jpg/117px-%22Good_bye%2C_sweetheart%22%2C_tobacco_label%2C_ca._1865.jpg');

var format = {
standard: {
internalFormat: 'RGB8',
format: 'RGB',
type: 'UNSIGNED_BYTE',
},
rgb565: {
internalFormat: 'RGB565',
format: 'RGB',
type: 'UNSIGNED_SHORT_5_6_5',
},
rgb9e5: {
internalFormat: 'RGB9_E5',
format: 'RGB',
type: 'FLOAT',
},
srgb: {
internalFormat: 'SRGB8',
format: 'RGB',
type: 'UNSIGNED_BYTE',
},
rgba32f: {
internalFormat: 'RGB32F',
format: 'RGB',
type: 'FLOAT',
},
rgba4: {
internalFormat: 'RGBA4',
format: 'RGBA',
type: 'UNSIGNED_SHORT_4_4_4_4',
},
rgb5a1: {
internalFormat: 'RGB5_A1',
format: 'RGBA',
type: 'UNSIGNED_SHORT_5_5_5_1',
},
luma: {
internalFormat: 'LUMINANCE',
format: 'LUMINANCE',
type: 'UNSIGNED_BYTE',
},
};

function compareFormatSize(image) {
return new Promise((r, _) => {
\t\tcreateImageBitmap(image, {
premultiplyAlpha: 'none',
colorSpaceConversion: 'none',
}).then(async bitmap => {
var text = String(image.src.match(/(?<=\\.)\\w{3,4}$/)).toUpperCase();
console.log(`${text === 'JPG' ? 'Colored jpg' : text}:`);
for (let val of Object.values(format)) {
await logBlobSize(bitmap, val);
\t\t\t\t if(val.format === 'LUMINANCE') r();
}
}).catch(console.warn);
});
};

compareFormatSize(jpeg).then(_ => compareFormatSize(png)).then(_ => compareFormatSize(jpgClr));

function logBlobSize(bitmap, { internalFormat, format, type }) {
return new Promise(r => {
drawCanvas(bitmap, internalFormat, format, type).convertToBlob({
type: `image/webp`
}).then(blob => { console.log(`Blob from ${internalFormat} is ${blob.size}b`); r(); });
})
}

function drawCanvas(bitmap, internalFormat, format, type) {
const gl = (new OffscreenCanvas(bitmap.width, bitmap.height)).getContext("webgl2", {
antialias: false,
alpha: false,
depth: false,
});

function createShader(gl, type, glsl) {
const shader = gl.createShader(type);
gl.shaderSource(shader, glsl)
gl.compileShader(shader);
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
console.error(gl.getShaderInfoLog(shader));
gl.deleteShader(shader);
return;
}
return shader;
}

const vs = createShader(
gl,
gl.VERTEX_SHADER,
`#version 300 es
#define POSITION_LOCATION 0
layout(location = POSITION_LOCATION) in vec2 position;

void main()
{
gl_Position = vec4(position, 0.0, 1.0);
}`,
);

const fs = createShader(
gl,
gl.FRAGMENT_SHADER,
`#version 300 es
precision mediump float;
precision mediump sampler2D;

uniform sampler2D sampler;
uniform vec2 dimensions;

out vec4 color;

void main()
{
color = texture(sampler, vec2(gl_FragCoord.x/dimensions.x, 1.0 - (gl_FragCoord.y/dimensions.y)));
}`,
);

const program = gl.createProgram();
gl.attachShader(program, vs);
gl.attachShader(program, fs);
gl.linkProgram(program);

const sampler = gl.getUniformLocation(program, 'sampler');
const dimensions = gl.getUniformLocation(program, 'dimensions');
const position = 0; // GLSL location

const vao = gl.createVertexArray();
gl.bindVertexArray(vao);
gl.enableVertexAttribArray(position);
const vxBuffer = gl.createBuffer();
const vertices = new Float32Array([
-1.0,-1.0,
1.0,-1.0,
-1.0, 1.0,
1.0, 1.0,
]);
gl.bindBuffer(gl.ARRAY_BUFFER, vxBuffer);
gl.vertexAttribPointer(position, 2, gl.FLOAT, false, 0, 0);
gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW);

const texture = gl.createTexture();
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, texture);
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, false);
gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false);
gl.pixelStorei(gl.UNPACK_COLORSPACE_CONVERSION_WEBGL, gl.NONE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texImage2D(
gl.TEXTURE_2D,
0,
gl[internalFormat],
bitmap.width,
bitmap.height,
0,
gl[format],
gl[type],
bitmap
);

gl.useProgram(program);
gl.uniform1i(sampler, 0);
gl.uniform2f(dimensions, gl.canvas.width, gl.canvas.height);
gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);

gl.deleteTexture(texture);
gl.deleteVertexArray(vao);
gl.deleteBuffer(vxBuffer);
gl.deleteProgram(program);

\treturn gl.canvas;
}
})()

提前致谢！

相关讨论

画布始终为RGBA 8位(32位颜色)。有人谈论添加选项以使画布更深以支持高清彩色显示，但尚未提供。

因此，调用canvas.converToBlob总是会给您RGBA32bit png(或jpeg)。创建LUMIANCE纹理将为您提供黑白纹理，但它会被绘制到RGBA 32位画布中。没有选择获取1通道PNG的选项。

至于RGB565，RGBA5551等这些格式可能受硬件直接支持或不被硬件直接支持，该规范允许驱动程序选择分辨率更高的格式，我想大多数台式机在您上传数据时会将数据扩展为RGBA8因此不会节省任何内存。

另一方面，WebGL规范要求以RGB565或RGBA5551的格式上传图像时，首先将图像转换为该格式，因此浏览器将获取图像并有效地将其量化为这些颜色深度，这意味着您失去色彩。然后，您将量化的图像绘制回画布上并保存，因此当然会压缩得更好，因为存在更多相似的颜色。

从WebGL规范中获取ImageBitmap的texImage2D版本

The source image data is conceptually first converted to the data type and format specified by the format and type arguments, and then transferred to the WebGL implementation. Format conversion is performed according to the following table. If a packed pixel format is specified which would imply loss of bits of precision from the image data, this loss of precision must occur.

让我们在没有WebGL的情况下尝试

(async() => {
function createImage(src) {
return new Promise((rs, rj) => {
const img = new Image();
img.crossOrigin = 'anonymous';
img.src = src;
img.onload = () => rs(img);
img.onerror = rj;
});
};

const jpeg = await createImage('https://upload.wikimedia.org/wikipedia/commons/a/aa/5inchHowitzerFiringGallipoli1915.jpeg');

const png = await createImage('https://upload.wikimedia.org/wikipedia/commons/2/2c/6.d%C3%ADl_html_m2fdede78.png');

const jpgClr = await createImage('https://upload.wikimedia.org/wikipedia/commons/thumb/e/ed/%22Good_bye%2C_sweetheart%22%2C_tobacco_label%2C_ca._1865.jpg/117px-%22Good_bye%2C_sweetheart%22%2C_tobacco_label%2C_ca._1865.jpg');

const format = {
standard: {
internalFormat: 'RGB8',
format: 'RGB',
type: 'UNSIGNED_BYTE',
fn: p => [p[0], p[1], p[2], 255],
},
rgb565: {
internalFormat: 'RGB565',
format: 'RGB',
type: 'UNSIGNED_SHORT_5_6_5',
fn: p => [
(p[0] >> 3) * 255 / 31,
(p[1] >> 2) * 255 / 63,
(p[2] >> 3) * 255 / 31,
255,
],
},
rgba4: {
internalFormat: 'RGBA4',
format: 'RGBA',
type: 'UNSIGNED_SHORT_4_4_4_4',
fn: p => [
(p[0] >> 4) * 255 / 15,
(p[1] >> 4) * 255 / 15,
(p[2] >> 4) * 255 / 15,
(p[3] >> 4) * 255 / 15,
],
},
rgb5a1: {
internalFormat: 'RGB5_A1',
format: 'RGBA',
type: 'UNSIGNED_SHORT_5_5_5_1',
fn: p => [
(p[0] >> 3) * 255 / 31,
(p[1] >> 3) * 255 / 31,
(p[2] >> 3) * 255 / 31,
(p[3] >> 7) * 255 / 1,
],
},
luma: {
internalFormat: 'LUMINANCE',
format: 'LUMINANCE',
type: 'UNSIGNED_BYTE',
fn: p => [p[0], p[0], p[0], 255],
},
};

async function compareFormatSize(image) {
const bitmap = await createImageBitmap(image, {
premultiplyAlpha: 'none',
colorSpaceConversion: 'none',
});
const text = String(image.src.match(/(?<=\\.)\\w{3,4}$/)).toUpperCase();
log(`${text === 'JPG' ? 'Colored jpg' : text}:`);
for (const val of Object.values(format)) {
await logBlobSize(bitmap, val);
}
};

await compareFormatSize(jpeg);
await compareFormatSize(png);
await compareFormatSize(jpgClr);

async function logBlobSize(bitmap, {
internalFormat,
format,
type,
fn,
}) {
const canvas = drawCanvas(bitmap, internalFormat, format, type);
const blob = await canvas.convertToBlob({
type: `image/webp`
});
const canvas2 = drawFn(bitmap, fn);
const blob2 = await canvas2.convertToBlob({
type: `image/webp`
});
log(`Blob from ${internalFormat} is ${blob.size}b(webgl) vs ${blob2.size}b(code)`);
if (false) {
const img = new Image();
img.src = URL.createObjectURL(blob);
document.body.appendChild(img);
const img2 = new Image();
img2.src = URL.createObjectURL(blob2);
document.body.appendChild(img2);
}
}

function drawFn(bitmap, fn) {
const ctx = (new OffscreenCanvas(bitmap.width, bitmap.height)).getContext("2d");
ctx.drawImage(bitmap, 0, 0);
const imageData = ctx.getImageData(0, 0, bitmap.width, bitmap.height);
const pixels = imageData.data;
for (let i = 0; i < pixels.length; i += 4) {
const n = fn(pixels.subarray(i, i + 4));
pixels.set(n, i);
}
ctx.putImageData(imageData, 0, 0);
return ctx.canvas;
}

function drawCanvas(bitmap, internalFormat, format, type) {
const gl = (new OffscreenCanvas(bitmap.width, bitmap.height)).getContext("webgl2", {
antialias: false,
alpha: false,
depth: false,
});

function createShader(gl, type, glsl) {
const shader = gl.createShader(type);
gl.shaderSource(shader, glsl)
gl.compileShader(shader);
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
console.error(gl.getShaderInfoLog(shader));
gl.deleteShader(shader);
return;
}
return shader;
}

const vs = createShader(
gl,
gl.VERTEX_SHADER,
`#version 300 es
#define POSITION_LOCATION 0
layout(location = POSITION_LOCATION) in vec2 position;

void main()
{
gl_Position = vec4(position, 0.0, 1.0);
}`,
);

const fs = createShader(
gl,
gl.FRAGMENT_SHADER,
`#version 300 es
precision mediump float;
precision mediump sampler2D;

uniform sampler2D sampler;
uniform vec2 dimensions;

out vec4 color;

void main()
{
color = texture(sampler, vec2(gl_FragCoord.x/dimensions.x, 1.0 - (gl_FragCoord.y/dimensions.y)));
}`,
);

const program = gl.createProgram();
gl.attachShader(program, vs);
gl.attachShader(program, fs);
gl.linkProgram(program);

const sampler = gl.getUniformLocation(program, 'sampler');
const dimensions = gl.getUniformLocation(program, 'dimensions');
const position = 0; // GLSL location

const vao = gl.createVertexArray();
gl.bindVertexArray(vao);
gl.enableVertexAttribArray(position);
const vxBuffer = gl.createBuffer();
const vertices = new Float32Array([-1.0, -1.0,
1.0, -1.0, -1.0, 1.0,
1.0, 1.0,
]);
gl.bindBuffer(gl.ARRAY_BUFFER, vxBuffer);
gl.vertexAttribPointer(position, 2, gl.FLOAT, false, 0, 0);
gl.bufferData(gl.ARRAY_BUFFER, vertices, gl.STATIC_DRAW);

const texture = gl.createTexture();
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, texture);
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, false);
gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false);
gl.pixelStorei(gl.UNPACK_COLORSPACE_CONVERSION_WEBGL, gl.NONE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texImage2D(
gl.TEXTURE_2D,
0,
gl[internalFormat],
bitmap.width,
bitmap.height,
0,
gl[format],
gl[type],
bitmap
);

gl.useProgram(program);
gl.uniform1i(sampler, 0);
gl.uniform2f(dimensions, gl.canvas.width, gl.canvas.height);
gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);

gl.deleteTexture(texture);
gl.deleteVertexArray(vao);
gl.deleteBuffer(vxBuffer);
gl.deleteProgram(program);

return gl.canvas;
}
})()

function log(...args) {
const elem = document.createElement('pre');
elem.textContent = [...args].join(' ');
document.body.appendChild(elem);
}

1	pre { margin: 0; }

Why setting format in gl.texImage2D to gl.LUMINANCE instead of gl.RGB makes the blob made out of the canvas only ~5% smaller in filesize?

我没有看到这些结果。在您的示例中，黑白图像的大小与RGB vs LUMIANCE相同。彩色图像变为1/2尺寸。但是，当然，取决于压缩算法是否将黑白32位图像压缩为比彩色32位图像小，因为在所有情况下，调用convertToBlob时画布都是32位。