Benchmarking JVMs

While playing around with jZenith I realized that I somehow ignored OpenJ9 up to now. So I wanted to see if it actually makes a difference for a small test case.

As I wrote basically the same app over and over again in jZenith to play with different integrations I wanted to see how it looks for the example app for the Redis Plugin.

I use hey nowadays for REST benchmarks, because they draw these nice histograms.

Just a couple of quick numbers for further reference (and for when it finally runs with GraalVM). All numbers from my Laptop (Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz, 12 GB of RAM)

openjdk version "11" 2018-09-25
OpenJDK Runtime Environment (build 11+24-Ubuntu-118.04)
OpenJDK 64-Bit Server VM (build 11+24-Ubuntu-118.04, mixed mode, sharing)

Startup time is fairly consistent around 1.7 seconds, with JVM startup adding 1 second (!) of overhead in warm-up with 60 seconds of requests, then:

hey -z 10s http://localhost:8080/user/e01afce1-cf1d-49ab-a78d-53e5ca1032ad

Summary:
Total: 10.0086 secs
Slowest: 0.0401 secs
Fastest: 0.0038 secs
Average: 0.0098 secs
Requests/sec: 5103.0285


Response time histogram:
0.004 [1] |
0.007 [254] |
0.011 [43529] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.015 [5071] |■■■■■
0.018 [1446] |■
0.022 [518] |
0.026 [143] |
0.029 [79] |
0.033 [31] |
0.036 [1] |
0.040 [1] |


Latency distribution:
10% in 0.0084 secs
25% in 0.0088 secs
50% in 0.0091 secs
75% in 0.0096 secs
90% in 0.0127 secs
95% in 0.0142 secs
99% in 0.0193 secs

So 5k requests per second is quite good I’d say.

Next OpenJ9:

openjdk version "11" 2018-09-25
OpenJDK Runtime Environment AdoptOpenJDK (build 11+28-201810022340)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.10.0-rc2, JRE 11 Linux amd64-64-Bit 20181002_42 (JIT enabled, AOT enabled)
OpenJ9 - e44c4716
OMR - 32df9563
JCL - e80f5bd084 based on jdk-11+28)

Startup is slightly faster (measured internally) but the VM overhead is more around 1.5 secs.

Summary:
Total: 10.0055 secs
Slowest: 0.0378 secs
Fastest: 0.0030 secs
Average: 0.0116 secs
Requests/sec: 4313.1229


Response time histogram:
0.003 [1] |
0.007 [8] |
0.010 [961] |■
0.013 [40454] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.017 [738] |■
0.020 [52] |
0.024 [817] |■
0.027 [111] |
0.031 [2] |
0.034 [5] |
0.038 [6] |


Latency distribution:
10% in 0.0105 secs
25% in 0.0109 secs
50% in 0.0112 secs
75% in 0.0117 secs
90% in 0.0124 secs
95% in 0.0132 secs
99% in 0.0226 secs

So in this scenario HotSpot is definitely faster, but I guess measuring against a nightly of OpenJ9 is actually not really fair. We will see how it performs against Graal.