This post is by P99 CONF speaker Gunnar Morling, Principal Software Engineer at Red Hat. It first appeared on his blog on August 4, 2021. To hear more from Gunnar and many more performance-minded open source developers, register for P99 CONF today.
Unit testing, for performance.
It’s with great pleasure that I’m announcing the first official release of JfrUnit today!
JfrUnit is an extension to JUnit which allows you to assert JDK Flight Recorder events in your unit tests. This capability opens up a number of interesting use cases in the field of testing JVM-based applications:
- You can use JfrUnit to ensure your application produces the custom JFR events you expect it to emit
- You can use JfrUnit to identify potential performance regressions of your application by means of tracking JFR events e.g. for garbage collection, memory allocation and network I/O
- You can use JfrUnit together with JMC Agent for whitebox tests of your application, ensuring specific methods are invoked with the expected parameters and return values
Getting Started With JfrUnit
JfrUnit is available on Maven Central (a big shout-out to Andres Almiray for setting up a fully automated release pipeline using the excellent JReleaser project!). If you’re working with Apache Maven, add the following dependency to your pom.xml file:
...
<dependency>
<groupId>org.moditect.jfrunit</groupId>
<artifactId>jfrunit</artifactId>
<version>1.0.0.Alpha1</version>
<scope>test</scope>
</dependency>
...
Alternatively, you can of course build JfrUnit from source yourself, as described in the project’s README file.
What is ModiTect?
JfrUnit is part of the ModiTect family of open-source projects. All the ModiTect projects are in some way related to Java infrastructure, such as the Java Module System, or JDK Flight Recorder. Besides JfrUnit, the following project are currently developed under the ModiTect umbrella:
|
With that dependency in place, the steps of using JfrUnit are the following:
- Enable the JFR event type(s) you want to assert against
- Run the application logic under test
- Assert the emitted JFR events
To make things more tangible, here’s an example that asserts the memory allocation done by a Quarkus-based web application for a specific use case:
@Test
@EnableEvent("jdk.ObjectAllocationInNewTLAB") // (1)
@EnableEvent("jdk.ObjectAllocationOutsideTLAB")
public void retrieveTodoShouldYieldExpectedAllocation() throws Exception {
Random r = new Random();
HttpClient client = HttpClient.newBuilder()
.build();
// warm-up (2)
for (int i = 1; i<= WARMUP_ITERATIONS; i++) {
if (i % 1000 == 0) {
System.out.println(i);
}
executeRequest(r.nextInt(20) + 1, client);
}
jfrEvents.awaitEvents();
jfrEvents.reset(); // (3)
// (4)
for (int i = 1; i<= ITERATIONS; i++) {
if (i % 1000 == 0) {
System.out.println(i);
}
executeRequest(r.nextInt(20) + 1, client);
}
jfrEvents.awaitEvents(); // (5)
long sum = jfrEvents.filter(this::isObjectAllocationEvent)
.filter(this::isRelevantThread)
.mapToLong(this::getAllocationSize)
.sum();
assertThat(sum / ITERATIONS).isLessThan(33_000); // (6)
}
- Enable the
jdk.ObjectAllocationInNewTLAB
andjdk.ObjectAllocationOutsideTLAB
JFR event types; on Java 16 and beyond, you could also use the newjdk.ObjectAllocationSample
type instead - Do some warm-up iterations so to achieve a steady state for the memory allocation rate
- Reset the JfrUnit event collector after the warm-up
- Run the code under test, in this case invoking some REST API of the application
- Wait until all the events from the test have been received
- Run assertions against the JFR events, in this case summing up all memory allocations and asserting that the value per REST call isn’t larger than 33K (the exact threshold has been determined upfront)
The general idea behind this testing approach is that a regression in regards to metrics like memory allocation or I/O — e.g. with a database — can be a hint for a performance degredation. Allocating more memory than anticipated may be an indicator that your application started to do something which it hadn’t done before, and which may impact its latency and through-put characteristics.
To learn more about this approach for identifying potential performance regressions, please refer to this post, which introduced JfrUnit originally.
Groovier Tests With Spock
Thanks to an outstanding contribution by Petr Hejl, instead of the Java-based API, you can also use Groovy and the Spock framework for your JfrUnit tests, which makes for very compact and nicely readable tests. Here’s an example for asserting two JFR events using the Spock integration:
class JfrSpec extends Specification {
JfrEvents jfrEvents = new JfrEvents()
@EnableEvent('jdk.GarbageCollection') // (1)
@EnableEvent('jdk.ThreadSleep')
def 'should Have GC And Sleep Events' {
when: // (2)
System.gc()
sleep(1000)
then: // (3)
jfrEvents['jdk.GarbageCollection']
jfrEvents['jdk.ThreadSleep'].withTime(Duration.ofMillis(1000))
}
}
- Enable the JFR event type(s) you want to assert against
- Run the application logic under test
- Assert the emitted JFR events
To learn more about the Spock-based approach of using JfrUnit, please refer to the instructions in the README.
For getting started with JfrUnit yourself, you may take a look at the jfrunit-examples repo, which shows some common usages the project.
Outlook
This first Alpha release is an important milestone for the JfrUnit project. Since its inception in the December of last year, I’ve received tons of invaluable feedback, and the project has matured quite a bit.
In terms of next steps, apart from further expanding and honing the API, one area I’d like to explore with JfrUnit is keeping track of and analysing historical event data from multiple test runs over a longer period of time.
For instance, consider a case where your REST call allocates 33 KB today, 40 KB next month, 50 KB the month after, etc. Each increase by itself may not be problematic, but when comparing the results from today to those of a run in six months from now, a substantial regression may have accumulated. For identifying and analysing such trends, loading JfrUnit result data into a time series database, or repository systems like Hyperfoil Horreum, may be a very interesting feature.
On a related note, John O’Hara has started work towards automated event analysis using the rules system of JDK Mission Control, so stay tuned for some really exciting developments in this area!
Last but not least, I’d like say thank you to all the folks helping with the work on JfrUnit, be it through discussions, raising feature requests or bug reports, or code changes, including the following fine folks who have contributed to the JfrUnit repository at this point: Andres Almiray, Hash Zhang, Leonard Brünings, Manyanda Chitimbo, Matthias Andreas Benkard, Petr Hejl, Sam Brannen, Sullis, Thomas, Tivrfoa, and Tushar Badgu. Onwards and upwards!
Read the Full P99 CONF Agenda
To see the full breadth of speakers at P99 CONF, check out the published agenda, and register now to reserve your seat. P99 CONF will be held October 6 and 7, 2021, as a free online virtual conference. Register today to attend!