Microbenchmarking Java image loading libraries

During my 3d engine project, the demand to load arbitrary image formats came up. My choice was ImageIO since it's shipped with Java. Short time ago, I realized that there is a cool Apache lib called commons imaging. The main goal would be to speed up the loading processes, so finally I have a reason to do some microbenchmarks, yey.

Since there are only very few tools for micro benchmarking and most of them offer poor features and documentation, I recommend to use JMH. As always, documentation and examples are kind of confusing, so here's the workflow I used.

First of all, you need two dependencies - the jmh core lib and the annotation processor.

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.9.1</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.9.1</version>
</dependency>

The methods you want to benchmark can then be annotated to be automatically registered for measurement.

@Benchmark
    public void commonsImage(){
        MainClass.loadImageCommonsImaging();
    }

Now a piece of software is needed that runs all your annotated methods. This can be done with some command line stuff, I prefer a solution that can be packaged as a jar or directly run from the IDE. Embedding your benchmark config in a class and write a small main method can do the job. The class you pass into the benchmark via the options is scanned for annotated methods.

public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(ImageLoadBenchmark.class.getSimpleName())
                .forks(1)
                .build();

        new Runner(opt).run();
    }

Run the main method from your IDE or export a package. Note: If you export a jar, you have to provide the dependencies - if you don't want them to reside in your classpath, create a fat jar with the maven-assembly-plugin. Tested it, works fine. Here's the result:

Benchmark         Mode  Cnt   Score   Error  Units
commonsImage  thrpt    20    8,127   0,077  ops/s
imageIOImage    thrpt    20   11,283  0,105  ops/s

So it seems as if imageIO can push more loading jobs per second than commons imaging. Dang it.

This is just a minimal benchmark setup - of course there's a ton of other things you could do with JMH. Have fun.