Introducing ZIO Http

On Mar 11th, 2021, ZIO Http was finally released at the ZIO World Conference, by Dream11. ZIO Http is a library that is used to build highly performant HTTP-based services and clients using functional scala and ZIO. We got a surprisingly overwhelming response from the scala community after its launch, which compelled us to write about the adventure we had building it.

Background

Dream11 relies on all the goodness of having a microservices-based architecture. Here is a glimpse of how it looks from 30,000 feet —

Between our clients (Android, iOS, and Web) and the service mesh, lies the critical “Edge Layer”. This is responsible for carrying out client aggregations on the backend and also works as an API gateway to our microservices.

Just to give a perspective of our scale, this layer can typically get around a million requests within a second during a sports match. That is a huge number and we deploy tens of thousands of servers to support this kind of scale in production. But this is not the most difficult part. For us, the complexity lies in the unpredictable nature of the traffic.

For a match where the traffic is floating at around a few thousand requests per sec, based on a player’s performance on the field, it can surge up to a million requests per second within a few minutes. This kind of surge renders “auto-scaling” practically useless and we have to provision for the maximum expected capacity at all times during the match. These kinds of surges make the edge infrastructure extremely sensitive to performance.

Motivation

For the last IPL season (Sept 2020) we wanted to add push-capabilities to our edge layer. This would enable the microservices to send messages to our clients over WebSockets.

Including client-integration, testing, release, and with some extra buffer time, we estimated that four weeks would be more than enough to implement this “simple” use case. All we had to do now was to select one web framework, which had decent WebSocket support.

Evaluation

However, it wasn’t that straightforward, surprise, surprise! We wanted our framework of choice to satisfy some criteria. It had to —

🎖 Be Battle Tested

The framework had to be stable and mature. We couldn’t afford to depend on something that’s bleeding edge, especially at our scale.

🧘 Support Functional Scala

The team was well versed with functional programming using scala and we had already seen massive productivity gains in other projects.

💪 Have ZIO Power

Considering WebSocket services are stateful, we wanted to leverage the power of ZIO for STM, Managed, and Stream.

🚀 Be Highly Performant

Needless to say, this was important considering how performance-sensitive our infrastructure was.

We had come up with the checklist and it seemed quite reasonable. There are a ton of frameworks out there that are battle-tested and support scala. And in the worst case, ZIO interoperability could be built-in, as an ad hoc capability. We were also sure that JVM would provide us with adequate performance for this use case. Overall, we were positive that within a day or two, we could finalize on the framework, and then the actual work could start. In the meantime, the client work had begun.

But, more than a week later, this was what the team came up with —

On one hand, we had Http4s, a battle-tested framework with solid functional backing, and some ZIO support using interop-cats, but it was the least performing framework in our tests. Vert.x, on the other hand, was great performance-wise, but focused on Java developers and didn’t bother about functional programming or ZIO. Finagle and Play fit somewhere in between and when we tried to add ZIO support to these frameworks, we observed a massive dip in performance. There was no clear winner in our evaluation.

The IPL season was fast approaching and we were already almost two weeks into our original estimate. Work had not started, we didn’t even have a usable framework, let alone the unknowns of using a new library in production, load testing, and client integration. And that’s when it hit us, it was going to be impossible to meet this deadline.

Building In-House

I look back now and think how in the world did we come up with this radical idea! After some serious consideration and much brain-storming, we decided to build a framework in-house. Dream11 has vast experience in building highly scalable systems. More importantly, building the framework in-house could reduce the probability of last-minute unknowns of using a new library so close to the season.

Since only two weeks were remaining, we put all hands on deck and started churning out the code. Mind you, due to the Covid19 pandemic, this was all being done remotely, while whiteboarding over calls, trying to come up with the best design to solve our use case.

ZIO Http

This is how ZIO Http was born, it was a bunch of hacky code glued together to solve only one use case — publishing messages to our clients using WebSockets reliably.

This is how the production code looked in our first release. It was barely 20 lines, with no ceremonious imports or DSLs. We pattern matched on the request and resolved it with either a health status or mount a ZIO-based WebSocket app. That was all! It was functional scala at its best, with native support for ZIO.

And yes, we had met our once seemingly impossible two-week deadline! There were zero outages and ZIO Http fared well in production even under high stress.

But, there was one major problem — the performance of our framework was ridiculously low. We were around 4x slower than Http4s which was already the slowest as per our benchmarks. This wasn’t surprising to us because we knew functional programming came with a considerable performance overhead. And, our primary goal with the first release was to meet the timelines.

Performance Focus

By mid-November 2020, the IPL was over and now, we started to think about performance. Beating Vert.x looked like a piped dream unless we dropped functional programming completely. We wanted to use functional scala and at least make ZIO Http comparable to Http4s by the next season (Apr 2021). Here’s how:

🚀 Low Hanging Fruits

We started by profiling and identifying hotspots in the execution path, and rewrote certain parts that were taking too long to execute. Though this was a good start, it could barely make any impact on performance.

🚀 Micro Optimizations

Next, we applied some standard micro-optimization techniques and reduced memory allocations along the way. These optimizations had a massive impact in terms of performance and we finally started becoming comparable to Http4s.

🚀 Thread Tuning

We looked at our thread pool usage. We were able to identify the kind of workload produced by the user program and intelligently decide which thread pool to execute in. This alone made a significant impact and we were now performing better than Http4s. But we didn’t stop here.

🚀 Rewrite Core

Not once, but multiple times, the guts were pulled out, re-engineered, and put back in. We dropped a few capabilities from the library in this process and hell, took down production a couple of times too!

All this was done with a single focus in mind, to improve the performance of the library. Week by week and bit by bit we extracted every quantum of performance that was possible from an HTTP library. The results we were able to achieve were astonishing!

ZIO-Http not only outperformed Http4s in our benchmarks, but it also blew past the rest of the frameworks including Vert.x, making it, objectively speaking — The fastest framework out there. This was insane!

Evolution

Though we started as a WebSocket-only framework, over the last few months we have been working continuously on adding more standard HTTP capabilities to the library, including an HTTP Client.

Today, the library is being used at Dream11 publishing messages to millions of users concurrently. It can achieve this using just 20 instances at peak traffic, without any issues and extremely predictable performance.

Going back to our original checklist, ZIO Http checked all our boxes. It was battle-tested at Dream11-scale, was officially the most performant library and all this was available using purely functional design with native support for ZIO.

Conclusion

ZIO Http is envisioned to be the de facto framework for any Http app written in scala. Like all open-sourced libraries, it needs contributions and feedback to achieve this goal. If you do functional scala in the large and don’t want to compromise on performance, ZIO Http is the library for you. Do check it out, share your feedback on discord and please star the repository to show your support.