When decoding the file it is different(?) from what it supposed to be

[nyelmer]

Hello, I am trying to understand why the encoded data bytes are different from what I encoded.

For instance, I am generating an image with starting values (as RGBA )[0, 0, 85, 255, ...] and this should be somewhat dark purpe-ish colour. However, it is shown as a bright teal-blue colour. When I inspect the decoded result I see that instead of [0, 0, 85, 255, ...] it is coded as [8, 78, 255, 255, ...]. Am I missing something about the encoding/decoding process?

I tried to play with colorTransform, formatAsRGBA, tolerantDecoding when decoding but, the result did not differ much.

If it helps here is my full data:

Encoded Data:
[0,0,85,255,0,0,50,255,0,0,70,255,0,0,115,255,0,0,100,255,0,0,71,255,0,0,86,255,0,0,107,255,0,0,88,255,0,0,49,255,0,0,47,255,0,0,110,255,0,0,108,255,0,0,69,255,0,0,86,255,0,0,116,255,0,0,82,255,0,0,102,255,0,0,87,255,0,0,103,255,0,0,88,255,0,0,87,255,0,0,118,255,0,0,57,255,0,0,69,255,0,0,70,255,0,0,69,255,0,0,107,255,0,0,76,255,0,0,49,255,0,0,65,255,0,0,113,255,0,0,122,255,0,0,118,255,0,0,111,255,0,0,102,255,0,0,51,255,0,0,57,255,0,0,77,255,0,0,90,255,0,0,81,255,0,0,110,255,0,0,115,255,0,0,61,255,0,0,48,255,0,0,48,255,0,0,48,255,0,0,48,255,0,0,48,255]

Decoded Data:
[0,77,251,255,0,71,239,255,0,76,237,255,0,91,251,255,8,97,255,255,7,88,255,255,3,75,255,255,9,101,255,255,0,90,255,255,0,86,250,255,0,93,254,255,6,97,255,255,1,90,255,255,0,79,255,255,13,102,255,255,0,88,255,255,0,79,246,255,0,84,245,255,0,90,248,255,0,88,247,255,0,86,248,255,1,85,255,255,0,73,250,255,0,68,240,255,0,78,245,255,0,92,251,255,0,98,252,255,0,103,252,255,0,82,255,255,0,70,249,255,0,68,249,255,8,83,255,255,19,99,255,255,16,108,255,255,9,115,255,255,0,86,255,255,0,71,250,255,0,63,251,255,10,71,255,255,18,81,255,255,12,87,255,255,2,91,255,255,1,94,255,255,0,72,249,255,0,52,245,255,1,48,254,255,6,50,255,255,0,52,254,255,0,59,247,255]

Any help would be much appreciated.

[videetparekh]

+1 on this issue.

Issue or Feature

I'm trying to recreate the pixel data of an JPEG image from base64 or URL and compare it to the pixel values generated in Python's Pillow library, which is common regarded as a solid benchmark for Image Processing in Python. I noticed that an image read by jpeg-js differs significantly from the same image in Python. Here are the scripts I've used to visualize this. I thought to compare node-canvas and python to jpeg-js as well, and found that all three of them differ.

Steps to Reproduce

Script used to generate a random image and then read and convert to pixel data (generates a comma-separated pixel file).

import numpy as np
from PIL import Image

arr = np.zeros((32,32,3), dtype=np.uint8)

for i in range(32):
  for j in range(32):
    arr[i,j] = np.random.randint(255, size=(1,3))

im = Image.fromarray(arr)
im.save('test.jpg')

img_8u = np.array(Image.open('test.jpg').getdata(), dtype=np.uint8)
with open('img_py.txt', 'w') as f:
  f.write(','.join(map(str, img_8u.flatten().tolist())))

Script used to generate pixel data in Nodejs (generates a comma-separated pixel file):

const Canvas = require("canvas")
const fs = require("fs")
const jpeg = require("jpeg-js")

function drawCanvas(url, shape) {
    var cvs = Canvas.createCanvas(shape[0], shape[1]);
    var context = cvs.getContext('2d');
    var img = new Canvas.Image;
    img.onload = function() {
        context.drawImage(img, 0, 0);
    };
    img.src = url;
    return context;
}

function preprocessAndStore(imageData, fileName) {
    var rgbFP32 = new Float32Array(cleanAndStripAlpha(imageData))
    fs.writeFile(fileName, rgbFP32.join(','), (err) => {
        if (err) throw err;
    })
}

function cleanAndStripAlpha(imageData) {
    const width = imageData.width;
    const height = imageData.height;
    const npixels = width * height;

    const rgbaU8 = imageData.data;

    // Drop alpha channel and retain rgb
    const rgbU8 = new Uint8Array(npixels * 3);
    for (let i = 0; i < npixels; ++i) {
        rgbU8[i * 3] = rgbaU8[i * 4];
        rgbU8[i * 3 + 1] = rgbaU8[i * 4 + 1];
        rgbU8[i * 3 + 2] = rgbaU8[i * 4 + 2];
    }
    return rgbU8;
}

// Node-Canvas
// var width = 32
// var height = 32
// var imgData = drawCanvas("test.jpg", [width, height])
// preprocessAndStore(imgData.getImageData(0, 0, width, height), 'image_canv.txt')


// JPEG JS
var jpegdata = fs.readFileSync('test.jpg')
var rawImgData = jpeg.decode(jpegdata)
preprocessAndStore(rawImgData, 'image_jpegjs.txt')

The two text files should be comparable with any diff checker. If you wish to recreate all my experiments, you will find that monochrome images or images with a uniform color will be regenerated correctly but the moment variation is introduced (an image of a dog for example), the colors returned are different, even if the image may look similar.

What might be causing this and how can this be resolved?

Your Environment

• Version of jpeg-js: 0.4.1
• Environment (e.g. node 4.2.0 on Mac OS X 10.8): node v12.16.1

[patrickhulce]

Thanks for filing!

For an encoding-decoding loop not being the same, those pixel values do seem very far off. If you're able to track down whether it's the encoder or the decoder at fault that'd be a good start to hunt down the bug. Does it appear bright-blue teal in other programs that can view JPEGs or just the values decoded by jpeg-js?

For the decoding not matching other libraries, my best guess is that the adverse affects of speed-conscious shortcuts taken by jpeg-js for inverse DCT math are popping up for a particular combination of patterns. If you have a case where it's waaaay off for a particular image, we can try to add a test case of it here and we'd happily accept a PR to fix, but I wouldn't be optimistic anyone can jump on hunting it down. Tracking down the source of bugs in the core of a 10-year old JPEG decoding library in JS is a very labor-intensive process.

[damoclark]

Hi @patrickhulce and @videetparekh

I have come into the same situation wrt different decoding of jpg images with jpeg-js and PIL. When I compare the bytes for each band for each pixel, sometimes they are off by a value of 1, and sometimes they are not. And it can be any of the RGB bands that are out.

It almost seems like there is a rounding difference happening somewhere, although when I had a quick look at the code for jpeg-js, it appears to be all bit-arithmetic. I took a quick look at Pillow for Python, and the decoding is implemented in C. I got as far as the JpegDecode.c file, but couldn't really ascertain why it is different.

The difference aren't perceptible to the eye, but my tensorflow object detection output is different between the decoding from each of these two libraries. It's a little puzzling.

[videetparekh]

I've not made any headway here. My understanding is, most libraries and languages (C++, Java, Python, JS Native), use C libs under the hood. So there's a single source of truth (libjpeg).

For a pure JS decoder, one has to create a brand new encoder/decoder, which may not match with the work libjpeg does, since the JPEG standard is vast and supports multiple conflicting decode methods(?).

TL;DR, the internet of strangers suggests that the decoding is "right" under the JPEG standard, but obviously skews results for an ML approach. Your best bet would be to try to use a Native npm library that uses C under the hood.

[damoclark]

No worries @videetparekh - I understand.

[patrickhulce]

When I compare the bytes for each band for each pixel, sometimes they are off by a value of 1, and sometimes they are not. And it can be any of the RGB bands that are out.

FWIW, this is mostly to be expected for the choices made in jpeg-js and not something we'll ever be trying to fix. Multiple color channels off by 8+ like the OP reports are what's within scope of a possible fix.

The difference aren't perceptible to the eye, but my tensorflow object detection output is different between the decoding from each of these two libraries. It's a little puzzling.

Sounds like some classic overfitting if the prediction differences are large ;) minor threshold changes are always unfortunate, but somewhat unavoidable here. See sharp recommendation below

For a pure JS decoder, one has to create a brand new encoder/decoder, which may not match with the work libjpeg does, since the JPEG standard is vast and supports multiple conflicting decode methods(?).

Exactly :)

Your best bet would be to try to use a Native npm library that uses C under the hood.

Yes, a million times, yes! If you have access to native modules (i.e. you're just using node), DO NOT use jpeg-js, use sharp instead. jpeg-js should only be used in situations where a pure JS implementation is required.

[damoclark]

Hey @patrickhulce

FWIW, this is mostly to be expected for the choices made in jpeg-js and not something we'll ever be trying to fix. Multiple color channels off by 8+ like the OP reports are what's within scope of a possible fix.

Yes, I understand.

Yes, a million times, yes! If you have access to native modules (i.e. you're just using node), DO NOT use jpeg-js, use sharp instead. jpeg-js should only be used in situations where a pure JS implementation is required.

I was returning to this issue to explain that I found a great native library called sharp that fits my purposes nicely. :) I compared the decoding with that of Pillow and it is identical.

And its super fast.

Appreciate you both taking the time to respond.

Cheers,
Damien.