Add automatic image sharpening

[janko]

Images pretty often get a little blurry when resized, so programs such as Photoshop will often apply some sharpening afterwards to make the images a little crisper. I think we should do the same here (continuing the discussion from https://github.com/janko-m/image_processing/issues/16#issuecomment-374677906).

ImageMagick

From this article it seems that for ImageMagick we should use the following:

unsharp "0.25x0.25+8+0.065"

libvips

For libvips, the vips_sharpen() documentation recommends the following defaults:

sharpen(sigma: 0.5, x1: 2, y2: 10, y3: 20, m1: 0, m2: 3)

The carrierwave-vips and vips-process gems both use a "sharpen mask" which they apply using vips_conv(). I'm copy-pasting @mokolabs' port to the newer libvips version from https://github.com/janko-m/image_processing/issues/16#issuecomment-375037420:

mask = Vips::Image.new_from_array [
       [-1, -1, -1],
       [-1, 24, -1],
       [-1, -1, -1]], 16

conv(mask)

@mokolabs Any ideas which ones of these two we should use? I'm leaning towards using vips_sharpen() as that's what it was designed for. But if it produces worse results than the vips_conv() approach then we might need to tweak the parameters. Hopefully the parameters should be fine if it's recommended in the documentation.

[janko]

Also, since this is tightly related to resizing, I think this should be done automatically after one of the resize_* methods. And these methods should accept a :sharpen option to allow modifying the parameters, e.g. sharpen: { sigma: 1.0 } (btw, I think the given parameters should be merged with the default ones). The user should also be able to disable sharpening with sharpen: false.

Also, as @mokolabs suggested in https://github.com/janko-m/image_processing/issues/16#issuecomment-375020142, we should probably do this automatically only for JPEG images.

[mokolabs]

@janko-m For the libvips implementation, I'd recommend sticking with the mask-style sharpening method using vips_conv.

The libvips vips_sharpen() method was designed for sharpening print images. While the settings above do sharpen screen images, they're not easily understood or modified. After hours of reading source code and testing sample images, I am still confused by those settings. Want more or less sharpening? Good luck!

By contrast, the mask-style sharpening method using vips_conv is easily understood and tweaked since using a matrix transformation on image data is a well-established practice. Plus, since this approach is already in use by the carrierwave-vips and vips-process gems, anyone using or contributing to those gems could migrate to image_processing more easily -- which would align well with the goals of this project.

That said, we should definitely strive for a great, out-of-the-box sharpen on images resized with either imagemagick and libvips -- so that no configuration is needed. I'm sooooooooo tired of thinking about these things. Let's just pick some awesome defaults and move on. 👍

(I'm happy to do a bit more testing and research on the image matrix settings, so that we're using the best practice thinking on those).

[janko]

After hours of reading source code and testing sample images, I am still confused by those settings. Want more or less sharpening? Good luck!

This is what the documentation suggests:

If you want more or less sharpening, we suggest you just change the m2 parameter.

But I agree the parameters are not intuitive. In contrast, ImageMagick's -unsharp has much more understandable parameters.

By contrast, the mask-style sharpening method using vips_conv is easily understood and tweaked since using a matrix transformation on image data is a well-established practice.

Agreed. I'm just a bit confused, the matrix that carrierwave-vips and vips-process use doesn't match any of the provided matrices in the Wikipedia article. Do you know how it originated? I'm asking because I would like "magic constants" to have an explanation in the comments.

Plus, since this approach is already in use by the carrierwave-vips and vips-process gems, anyone using or contributing to those gems could migrate to image_processing more easily -- which would align well with the goals of this project.

Agreed!

That said, we should definitely strive for a great, out-of-the-box sharpen on images resized with either imagemagick and libvips -- so that no configuration is needed.

I was looking now at some other projects I know that use libvips. Sharp applies this mask using vips_conv() by default, but when users modify the parameters it switches to vips_sharpen(). The VIPS author also seems to suggest the same matrix in h2non/imaginary#29 (comment). imgproxy doesn't seem to use any sharpening at the moment.

From this information I would suggest that we use the same mask as the sharp, as it's the most popular project that uses libvips. @mokolabs Can you see any visual or performance difference between sharp's

mask = Vips::Image.new_from_array [
       [-1, -1, -1],
       [-1, 32, -1],
       [-1, -1, -1]], 24

image.conv(mask)

and carrierwave-vips' & vips-process'

mask = Vips::Image.new_from_array [
       [-1, -1, -1],
       [-1, 24, -1],
       [-1, -1, -1]], 16

image.conv(mask)

If not, I would go with sharp's mask, as those folks really seem to know what they're doing 😃, and there is a large userbase that tested that mask.

[mokolabs]

Okay, I did some more testing and research.

• You can see a visual difference between 32/24 and 24/16

• 32/24 is slightly less sharp, but preserves image contrast

• 24/16 is definitely more sharp, but increases image contrast

32/24 is probably a better default choice since you're getting a reasonable amount of sharpening, but still preserving the contrast of the original image.

24/16 is sharper and shows more detail, but at a cost of increasing image contrast.

So, yeah, I think going with Sharp's mask is a totally reasonable choice.

...

FYI, here's an interesting graphic from this paper that shows how an unsharp matrix mask works:

So, in our case, if we replace 9 with 24 and set α to 1, the center number will be 32 (or 24 + (8 * 1)).

[mokolabs]

Also... this site has an interactive image matrix where you can play with the numbers in an image matrix and see how they change an image.

If you increase the center number to be higher than the divisor, it will lighten the overall image. If you then add negative numbers to the edges of the matrix, you will see that the negative numbers are darkening edges (and increasing contrast) and then the center number is lightening the image to make up for the increased darkness.

And that's basically how the unsharp matrix works. 😄

http://beej.us/blog/data/convolution-image-processing/

[janko]

@mokolabs Thank you so much for all this awesome research and information! ❤️

I'll go ahead and start working on the sharpening implementation.

[mokolabs]

No problem, @janko-m. It's not exactly fun, but I've learned sooooo much about image processing... and I'm excited to see improvements to this gem.

BTW, I think we might want to use a different default for Imagemagick sharpening.

Imagemagick has two basic ways of sharpening:

• -sharpen supports radius x sigma ("0x1")
• -unsharp supports radius x sigma + gain + threshold ("0.25x0.25+8+0.065")

In my testing, using the exact commands and values above, the -sharpen option looks way better.

Resize with no sharpening
convert black1.jpg -resize 1000x1000 hmmm1.jpg

Resize with -sharpen set to 0x1
convert black1.jpg -resize 1000x1000 -sharpen 0x1 hmmm2.jpg

Resize with -unsharp set to 0.25x0.5+8+0.065
convert black1.jpg -resize 1000x1000 -unsharp 0.25x0.5+8+0.065 hmmm3.jpg

Notice how edges are very jaggy and have more noise than the original.

Resize with -unsharp set to 0x1+1+0.065

This version is a lot better. By reducing the radius and sigma to match the 2nd test image, and also reducing the gain to 1, we get very close to the 2nd test image. But the 2nd image brings out a bit more detail in highlights.

Summary

FWIW, I'd probably still go with -sharpen option because it's simpler to use. The results look great and, with just two variables to tweak, it's easier to change. Just increase or decrease the sigma for more or less sharpening.

[janko]

@mokolabs Awesome, totally agreed, we'll go with -sharpen then for ImageMagick.

[semarillex]

@janko-m, is there any config option for disabling automatic sharpening for entire pipeline?

For example: ImageProcessing::MiniMagick.source(image).auto_sharpening(false)

[janko]

@volodymyr-shevchuk No, there currently isn't one. Since this would only concern the #resize_ methods, I don't feel a global setting like this would make sense, and it would also require modifying the internal design.

You can always DRY up passing sharpen: false to #resize_ operations, for example:

VERSION_DEFINITIONS = {
  large:  [:resize_to_limit, 800, 800],
  medium: [:resize_to_limit, 500, 500],
  small:  [:resize_to_limit, 300, 300],
  square: [:resize_to_fill,  150, 150],
}

versions = {}
VERSION_DEFINITIONS.each do |name, (operation, *args)|
  versions[name] = pipeline.send("#{operation}!", *args, sharpen: false)
end

[janko]

@mokolabs I just saw a conference talk where the presenter mentioned that without auto-sharpening 30% of their images were blurry (they had migrated from Shrine to Active Storage 5.2, which didn't yet use ImageProcessing). So, thanks again for all your help on adding it!

[maxence33]

Hello @janko @mokolabs , I have transitionned to VIPS and was trying to add sharpening that would mimick ImageMagick default sharpening which gave good results.

I was going with the default matrix:
[-1, -1, -1],
[-1, 32, -1],
[-1, -1, -1]
], 24

But it didn't add enough sharpening, especially to small images.
Also I have tried to tweak the figures in the matrix but couldn't get close to anything related to sharpening.
It either changed exposure, or contrast.

Thanks to @mokolabs link : http://beej.us/blog/data/convolution-image-processing/ I have kinda understood how to add strength to the image sharpening

Let's say we know :
s = strength of sharpening. (I have tested between 1 and 4.)
d = divider

And want to know :
r = resulting value

The matrix looks like :

[-s, -s, -s],
[-s, r, -s],
[-s, -s, -s]
], d

The missing value r is then equal to Σs + d

Regarding the d value, the link recommends to use the number of dots in the matrix, then switched the divider to 9.

Also I have noticed a gain in performance by computing the resulting pixel from only a subset of neighbours (and result is still consistent eventhough slightly less sharpened in high contrast zones)

[0, -s, 0],
[-s, r, -s],
[0, -s, 0]
], d

(Also Vips behavior was really strange: performance spiked to 25 seconds per image after a few matrix trials. Had to kill Puma, Sidekiq, and close the Chrome tab and reopen localhost in a new tab to see better performance again. Not sure where the problem cam from..)

I am really noob in Ruby (I do Rails Frankenstein code) and can't really help but that would be great to be able to supply the s value only, and image_processing to rebuild the matrix ? And try to find equivalent result for ImageMagick so that we don't have to tweak sharpening settings when switching from a processor to the other.

I am will still try to read image_processing code and see if I understand and maybe do a pull request .. but Janko will have to proofread me 😄

[janko]

Thank you for researching this, it sounds like you had interesting findings.

ImageProcessing currently allows you to override the convolution mask:

def sharpen_mask(s)
  d = 9
  r = 8 * s

  matrix = [[0, -s,  0],
            [-s, r, -s],
            [0, -s,  0]]

  Vips::Image.new_from_array matrix, d
end

ImageProcessing::Vips
  .source(image)
  .resize_to_limit!(600, 600, sharpen: sharpen_mask(1))

I would be open to supporting options for automatically building convolution masks based on the knowledge you posted, to help people adjust sharpening without having to understand convolution masks. I guess something like this would make sense:

.resize_to_limit!(600, 600, sharpen: { strength: 2 })

Also Vips behavior was really strange: performance spiked to 25 seconds per image after a few matrix trials. Had to kill Puma, Sidekiq, and close the Chrome tab and reopen localhost in a new tab to see better performance again. Not sure where the problem cam from..

Did that happen only with the convolution masks with zeroes in it, or even with the default one?

And try to find equivalent result for ImageMagick so that we don't have to tweak sharpening settings when switching from a processor to the other.

Yeah, that'd be really nice.

[maxence33]

Hi Janko,

Then if ImageMagick allows convolution masks conversion I guess we have a very similar way of sharpening for both IM and Vips. I have not tried the convolution mask in my Ubuntu console with IM yet but I don't see a reason why it should be different.

I will try to do a few tests by Sunday with IM and VIPS and see if we get consistent results between the 2 processes, for a similar sharpening matrix.

Your sharpen_mask function seems perfect. (Though with a matrix with 4 s, I guess r is rather something like r = 4 * s + d)

What I have noticed also is that a matrix with 8 s (well, I mean 8 cells like -s) makes a stronger sharpening than a matrix with 4 s. (r value is then r = 8 * s + d)
Then the matrix with 4 s allows for a smaller incremental step in sharpening. So I guess it is a better option with 4 s like you did in the function.

Yet a sharpening strength of 2, compared to 1, with a matrix with 4 s still makes a big difference.
For example my Shrine uploader looks like this at the moment :

sharpen_mask_2 = Vips::Image.new_from_array [
		  [0, -2, 0],
		  [-2, 17, -2],
		  [0, -2, 0]
		], 9

		sharpen_mask_1 = Vips::Image.new_from_array [
		  [0, -1, 0],
		  [-1, 13, -1],
		  [0, -1, 0]
		], 9

	    pipeline = ImageProcessing::Vips.source(original)
	    { 
	      large:  pipeline.saver(quality: 80, strip: true).resize_to_fit!(2000, 2000, sharpen: sharpen_mask_1),
	      small:  pipeline.saver(quality: 80, strip: true).resize_to_fit!(1200, 1200, sharpen: sharpen_mask_1),
	      thumb:  pipeline.saver(quality: 95, strip: true).resize_to_fit!(250, 250, sharpen: sharpen_mask_2)     
	    }

And it looks perfect.
I have increased sharpening for the thumb to the tune of 2, but anything above 3 is already overboard.

I am not sure if this is possible to modifiy the even refined further this incremental step in sharpness.. I tried to tweak the divisor also :

sharpen_mask_2 = Vips::Image.new_from_array [
		  [0, -2, 0],
		  [-2, 26, -2],
		  [0, -2, 0]
		], 18

		sharpen_mask_1 = Vips::Image.new_from_array [
		  [0, -1, 0],
		  [-1, 22, -1],
		  [0, -1, 0]
		], 18

=>I arbitrarily doubled up the divisor from 9 to 18 ..and VIPS processed 4 Shrine variants of a 13 MO jpg image in about 2 seconds which is consistent with what I used to have with the previous matrixes with a divisor of 9.
I can't see any difference in sharpening. The only difference is a slightly smaller variants : large variant went from 0.35MO to 0.33MO...
Then divisor does not seem to alter sharpness at all (to confirm maybe).

Even if we don't really have a way to refine the strength, setting s value with an integer (pull request 56) seems quite fulfilling. We can't get a more refined result by tweaking the matrix directly IMO..

Regarding the decrease of performance I got originally am not too sure.
I test through my app with Shrine, Uppy, Sidekiq.. locally. So any loss of performance is difficult to track. Also I made funny matrixes last time. Maybe it kinda broke VIPS for a moment.

Using the above matrixes don't really change performance over time on my side.

[maxence33]

Hi Janko,

I made a quick test with a single image sharpening with both IM and VIPS and it gave exactly the same results:
For VIPS sharpening I used my app with Shrine and Image_processing :

sharpen_mask_1 = Vips::Image.new_from_array [
		  [0, -4, 0],
		  [-4, 25, -4],
		  [0, -4, 0]
		], 9

pipeline = ImageProcessing::Vips.source(original)
	    { 
	      large:  pipeline.saver(strip: true).resize_to_fit!(6000, 6000, sharpen: sharpen_mask_1),	      
	    }

And locally with IM :
convert splash.jpg -define convolve:scale=0.11111111111111 -morphology Convolve '3x3: 0,-4,0 -4,25,-4 0,-4,0' new-splash.jpg

I used a divisor of 9 and a strength of 4.
Original image was 6000x4000. Resulting sizes for both was still 6000x4000.
Also in IM the divisor is actually written as a multiplicator : 1/9 or 0.111111111111

The resulting images were exactly the same. A sharpening with a strength of 4 was applied to both with no visible difference.

So using matrixes for sharpening should give exact same results whether we use VIPS or Im as processor.

[janko]

Nice that you found an ImageMagick alternative 👍

Does ImageMagick's -sharpen/-unsharp use convolution masks under the hood? If yes, is there any documentation how they translate to convolution masks? I believe these operators were added to hide the complexity of convolution masks, I would prefer to continue using them as they produce a simpler shell command.

According to this article, the gain parameter in -unsharp controls the amount of sharpening (in the article they call it amount). Could this be the "strength" parameter from our convolution mask?

Based on the current discussion and some research, I think it would make sense to have ImageProcessing::MiniMagick and ImageProcessing::Vips accept two types of sharpening options:

• options with meaningful names (e.g. "strength", "sigma" etc) – these would use ImageMagick's -unsharp and perhaps libvips' vips_sharpen() (see this comment for how vips_sharpen() compares to -unsharp), though it's probably better to keep using convolution matrices for libvips based on the prior discussion in this issue
• raw convolution matrices (array of arrays) + divisor – for libvips this would use vips_conv() as it does now, and for ImageMagick it would use the "convolve" options you just found

I don't know, that's just an idea. I feel that convolution matrices are complex for reading and seeing them in MiniMagick logs, but they seem to offer more control. I'm just wondering again whether we can use the simpler sharpening functions that ImageMagick/libvips already provide as default.

[maxence33]

I think having the two options is great.
Using Matrices allows for a higher level, and then obfuscate the processor.
Also it would be backwards compatible.

Regarding IM sharpening, there seem to be 2 functions:

• convert A_blur.jpg -sharpen 0x3 A_blur_sharp.jpg
  as described here : http://www.imagemagick.org/Usage/blur/
  where in -sharpen 0x3 0 is radius and 3 is sigma

As per the comments it seems Radius is the size of the matrice and Sigma the strength.
"It (ie Sigma) can be any floating point value from .1 for practically no sharpening to
3 or more for sever sharpening. 0.5 to 1.0 is rather good. "
It is quite matching my experience with the matrice: a strength of 4 in the matrice is already quite a strong sharpening.
(though the function probably translates into an 8 s matrice instead of a 4 s matrice that we have discussed earlier. The 4s matrice allowing smaller steps.)

Also the sharpen function allows fractional numbers for Sigma. As image_processing matrices is only using integers if I am not mistaken.

Regarding Radius, it can be set to 0. Though our smallest matrix is 3x3 which translates into a radius of 1 I guess. (1 pixel around the center pixel r). I am not sure what is set as radius behind the hood but in case Radius is set as 0, it should still at least translate into a matrice of 3x3. (unless my imagination is not broad enough)

• unsharp RadiusxSigma+Amount+Threshold

This function is even more complicate and explanation on this thread https://www.even.li/imagemagick-sharp-web-sized-photographs/ is not quite matching what we can read here http://www.imagemagick.org/Usage/blur/.
For example the strength of sharpening is said to be set by Amount as it is Sigma that is most affecting sharpening in the previous function.

Need to investigate on this.

Well I guess it is difficult to translate anything that is used at the moment : function for IM and matrice for Vips into something unified.

The best would be to have an inhouse higher-level single digit as sharpness strength that would overwrite the default sharpening. (well if it doesn't create problems elsewhere or in logs). That would be using matrices only, as we now know how to sharpen from matrices from both VIPS (we know that already) and for IM too (with -define convolve:scale= for the divisor and -morphology Convolve for the matrice)

And still allow a different, tweakable, sharpening method depending for each of the processors (function or matrice).

Then the user using image_processing for its "higher level" ability can add sharpening very easily.

[janko]

FYI, due to #67 (comment) and carrierwaveuploader/carrierwave#2481, I disabled default sharpening in MiniMagick backend in version 1.11.0.

[maxence33]

Thanks Janko. Actually I realise my comments deviated to discussing a higher level sharpening method which could both fit Vips and ImageMagick, with the help of the matrices.
But thanks for the heads up, I will check if some of my models are still relying on automatic sharpening, and add a matrice instead