This study compares the compression performance of four different image formats: JPEG, JPEG XR, WebP, and HEVC-MSP. The latter three formats were chosen because they are frequently discussed as possible JPEG successors.
It is our intent to only address compression performance in this study. Other technical, legal, and market factors that might be considered when evaluating codecs are outside the scope of this study.
We chose to test with four algorithms:
All of these algorithms compare two images and return a number indicating the degree to which the second image is similar to the first. In all cases, no matter what the scale, higher numbers indicate a higher degree of similarity.
It's unclear which algorithm is best in terms of human visual perception, so we tested with four of the most respected algorithms.
We had planned to include a set of very large images (~20 megapixels each) but some encoders had issues with them. Because this does not allow for a full comparison, the image set was cut for this study. Encoder developers were notified of any problems found.
All evaluation results should be easily reproducible using publicly available tools.
MATLAB is the only non-free (as in beer) software used. We recommend tweaking the test harness to use GNU Octave if you would like to test without access to MATLAB.
The following software is used to generate results for this study:
jpgcrush is used to further optimize the JPEG because
the JPEG encoder does not implement this lossless optimization.identify is used to extract width and height information from images, and convert
is used to convert between PNG and PPM formats. Both tools are part of the
ImageMagick tools. This study uses version 6.8.6-6.libjpeg is used to encode and decode JPEG images. This study uses version 9.
jxrlib is used to encode and decode JPEG XR images.
This study uses version 1.1.libwebp is used to encode and decode WebP images.
This study uses version 0.3.1.TAppEncoderStatic is used to encode HEVC-MSP bitstreams, and TAppDecoderStatic is used to decode HEVC-MSP bitstreams.
Both tools are part of the jctvc-hm software package.
This study uses r3531 of the SVN-based source code. See testing code to learn about invocation.ssim (Structural Similarity) is used to calculate an RGB-SSIM value between two PNG files. We use a
C++ implementation. We average the R, G, and B results. dump_psnrhvs is a C++ program, part of daala's tools, used to calculate a PSNR-HVS-M value between two Y'CbCr files. Daala git revision 4997e529e81bd3f50051fb84ff2aa6759b24caa6 was used.PNG test images are converted to CCIR 601 full-range Y'CbCr 4:2:0, which is then fed directly into the encoders. In order to convert back to PNG for quality scoring (e.g. SSIM) we decode to Y'CbCr 4:2:0 and then encode that to PNG. Doing this consistently allows us to avoid pre and post processing done by production encoders and decoders, and we test the encoding algorithms themselves as closely as possible. Direct encoding and decoding for JPEG, JPEG XR, and WebP is done via custom encoder and decoder programs (source code on github with the testing scripts) which call directly into the encoding and decoding APIs. The HEVC-MSP encoder and decoder accept and output Y'CbCr 4:2:0 directly, so no custom program is necessary.
HEVC-MSP files are penalized 80 bytes per image file because HEVC-MSP is just a bitstream with no container. This penalty approximates the size of container data.
The algorithm used, where F is the format being evaluated and Q is a JPEG quality level:
For image sets including multiple images, the result will be the arithmetic mean.
Note: Video formats such as VP8 and HEVC typically use 'studio swing' Y'CbCr with a restricted range of 16 to 235 instead of full range of 0 to 255. When working with RGB data the scaling for studio swing is accomplished as part of the colorspace conversion process. Some image formats derived from video formats, such as WebP, inherit video's conventional range in their common RGB conversions. In our study we adopted a methodology which uses identical colorspace conversion for all formats because the objective metrics were developed against greyscale images. While these metrics correlate well with perception in their intended applications they are known to exaggerate the perceptual impact of small brightness or contrast shifts that can be caused by differences in colorspace conversion. As a result the study does not consider the effect of colorspace or range difference that would typically be found in production, and manual visual spot checking did not suggest the conversion had a large effect on perceptual quality.
The following Excel (.xlsx) files contain the full results for this study. These can be opened with MS Excel or LibreOffice 4.1.x.
The goal for this section is to visualize file size ratios, where JPEG is always 1.0, over a range of JPEG qualities. File sizes are recorded at equivalent Y-SSIM values. In each graph, the Y axis represents file size ratios. The X axis represents a range of JPEG quality values. There is one graph for each image set.
|
 |
 |
The goal for this section is to visualize file size ratios, where JPEG is always 1.0, over a range of JPEG qualities. File sizes are recorded at equivalent RGB-SSIM values. In each graph, the Y axis represents file size ratios. The X axis represents a range of JPEG quality values. There is one graph for each image set.
 |
 |
 |
The goal for this section is to visualize file size ratios, where JPEG is always 1.0, over a range of JPEG qualities. File sizes are recorded at equivalent IW-SSIM values. In each graph, the Y axis represents file size ratios. The X axis represents a range of JPEG quality values. There is one graph for each image set.
 |
 |
 |
The goal for this section is to visualize file size ratios, where JPEG is always 1.0, over a range of JPEG qualities. File sizes are recorded at equivalent PSNR-HVS-M values. In each graph, the Y axis represents file size ratios. The X axis represents a range of JPEG quality values. There is one graph for each image set.
 |
 |
 |