Improving throughput and latency using Java Virtual Threads in Spring

One of the major changes that Java 21 brought about is Virtual threads. There is so much hype around it, but let’s see a real-world example with some metrics. Spring introduced the ability to create GraalVM native images that use Spring Boot and Java 21’s virtual threads(Project Loom).

We will look at virtual threads in detail and the many features that come with them in another article. We will focus on a simple project where we can enable the features of Java Virtual Threads and see the difference in performance gains.

Why virtual threads are faster than normal threads?

#

Normal threads in Java are tied to OS threads. So, there is a limitation to the actual number of threads it can create. Also, time is lost waiting for blocking I/O Calls. Normal Threads are called Platform Threads. They are an expensive resource that needs to be managed well.

Now, when it comes to virtual threads, they are lightweight constructs. Multiple virtual threads can be tied to a single platform thread which in turn gets tied to an OS thread.

The simple idea behind Virtual Threads

The Spring Project used for benchmarking

#

The project used in this article can be found here. It has a simple fetch API that returns the Thirukkural based on the ID sent.

Now, by default the http server in Tomcat can run many threads in parallel and this will not let us test out this feature easily. So let’s throttle it to 10 max threads by adding the below property in the application.properties file. This will allow Tomcat to only use 10 threads at max.

server.tomcat.threads.max=10

We will mimic a blocking IO call using sleep. We will also log the current thread it’s using.

Thread.sleep(1000);
log.info("Running on " + Thread.currentThread());

Benchmarking using hey

#

We will benchmark this using hey tool.

hey -n 200 -c 30 http://localhost:8080/thirukural/1

Summary:
 Total:        36.1759 secs
 Slowest:      8.0463 secs
 Fastest:      2.0080 secs
 Average:      5.6840 secs
 Requests/sec: 4.9757


Response time histogram:
 2.008 [1]     |
 2.612 [10]    |■■■
 3.216 [0]     |
 3.819 [0]     |
 4.423 [11]    |■■■
 5.027 [0]     |
 5.631 [0]     |
 6.235 [156]   |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
 6.839 [0]     |
 7.442 [0]     |
 8.046 [2]     |■


Latency distribution:
 10% in 4.0305 secs
 25% in 6.0212 secs
 50% in 6.0263 secs
 75% in 6.0340 secs
 90% in 6.0376 secs
 95% in 6.0387 secs
 99% in 8.0463 secs

Details (average, fastest, slowest):
 DNS+dialup:   0.0007 secs, 2.0080 secs, 8.0463 secs
 DNS-lookup:   0.0005 secs, 0.0000 secs, 0.0041 secs
 req write:    0.0000 secs, 0.0000 secs, 0.0003 secs
 resp wait:    5.6832 secs, 2.0079 secs, 8.0419 secs
 resp read:    0.0001 secs, 0.0000 secs, 0.0005 secs

Status code distribution:
 [200] 180 responses

A few log entries are listed below to see that these requests are using Platform Threads.

2024-06-30T22:07:29.659+05:30  INFO 2120 --- [thriukural] [nio-8080-exec-7] p.v.thriukural.web.ThirukuralController  : Running on Thread[#45,http-nio-8080-exec-7,5,main]
2024-06-30T22:07:29.659+05:30  INFO 2120 --- [thriukural] [io-8080-exec-10] p.v.thriukural.web.ThirukuralController  : Running on Thread[#48,http-nio-8080-exec-10,5,main]
2024-06-30T22:07:29.659+05:30  INFO 2120 --- [thriukural] [nio-8080-exec-2] p.v.thriukural.web.ThirukuralController  : Running on Thread[#40,http-nio-8080-exec-2,5,main]

Now, to enable the application to use virtual threads, we just add the below property in application.properties.

spring.threads.virtual.enabled=true

Let’s run and see the same benchmark results again.

hey -n 200 -c 30 http://localhost:8080/thirukural/1

Summary:
 Total:        12.2478 secs
 Slowest:      2.1747 secs
 Fastest:      2.0086 secs
 Average:      2.0410 secs
 Requests/sec: 14.6965


Response time histogram:
 2.009 [1]     |
 2.025 [149]   |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
 2.042 [0]     |
 2.058 [0]     |
 2.075 [0]     |
 2.092 [0]     |
 2.108 [0]     |
 2.125 [0]     |
 2.141 [0]     |
 2.158 [0]     |
 2.175 [30]    |■■■■■■■■


Latency distribution:
 10% in 2.0115 secs
 25% in 2.0129 secs
 50% in 2.0158 secs
 75% in 2.0188 secs
 90% in 2.1724 secs
 95% in 2.1739 secs
 99% in 2.1747 secs

Details (average, fastest, slowest):
 DNS+dialup:   0.0015 secs, 2.0086 secs, 2.1747 secs
 DNS-lookup:   0.0014 secs, 0.0000 secs, 0.0093 secs
 req write:    0.0000 secs, 0.0000 secs, 0.0002 secs
 resp wait:    2.0392 secs, 2.0085 secs, 2.1656 secs
 resp read:    0.0003 secs, 0.0000 secs, 0.0026 secs

Status code distribution:
 [200] 180 responses

A few log entries to see them running in virtual threads.

2024-06-30T22:12:38.485+05:30  INFO 17700 --- [thriukural] [mcat-handler-48] p.v.thriukural.web.ThirukuralController  : Running on VirtualThread[#103,tomcat-handler-48]/runnable@ForkJoinPool-1-worker-6
2024-06-30T22:12:38.485+05:30  INFO 17700 --- [thriukural] [mcat-handler-66] p.v.thriukural.web.ThirukuralController  : Running on VirtualThread[#121,tomcat-handler-66]/runnable@ForkJoinPool-1-worker-12
2024-06-30T22:12:38.485+05:30  INFO 17700 --- [thriukural] [mcat-handler-69] p.v.thriukural.web.ThirukuralController  : Running on VirtualThread[#124,tomcat-handler-69]/runnable@ForkJoinPool-1-worker-7

Benchmarking results

#

As evident from the results, we see an increased throughput and better latency as well once virtual threads are enabled.

Summary

#

A simple parameter change has enabled us with a lot of improved performance and throughput. We will further explore in future articles some of the considerations that need to be taken before using virtual threads.