Any considerations about migrating from Netty to Akka HTTP?

[kromenak]

Hello! We have a Play application that has been in production since 2017. Since then, we have used the Netty server, which I believe was the default (or maybe only option) at that time.

For an upcoming release, we have updated from Play 2.7.1 to Play 2.8.19. As part of that, since the default server backend is now Akka HTTP, we were planning to also migrate from Netty to Akka HTTP.

However, we are admittedly not super well versed in the differences between Netty and Akka HTTP. I suppose our worst fear would be that Akka HTTP is not as efficient or effective as Netty, or it doesn't have the same scaling characteristics. For example, I saw an older post on the Lightbend forums where a user was having some issues scaling Akka HTTP, and was ultimately able to fix it by manually changing the configuration for concurrent users (https://discuss.lightbend.com/t/scaling-play-2-7-x-with-akka-http-server-for-a-large-number-of-concurrent-sse-connections/7679/8).

Anyway, I guess the questions I had related to this:

• Can anyone speak to how comparable Netty and Akka HTTP are as drop-ins for one another? Are they configured in comparable ways to achieve similar levels of performance and scaling without requiring manual configuration?
• Does anyone have anecdotes or data about how migrating from Netty to Akka was, and whether any issues were encountered in production?
• Though Akka HTTP is now the default for Play, is it generally recommended to migrate? Is Akka HTTP generally considered to be more stable, performant, or to have better characteristics than Netty for Play applications?
• Any things we should consider or be concerned about when we decide to migrate from Netty to Akka?

Any insight or feedback would be greatly appreciated!

[ihostage]

Hi, @kromenak!
Sorry, I can't write a detailed answer, but my suggestion to you: still to use Netty.

Firstly, because we plan migrate from Akka HTTP to Pekko HTTP in Play 3. You can get more information in this PR playframework#11804. It hasn't merged yet, but it have a lot of things about current context.

Secondly, we planed but didn't make Netty as default backend in Play 2.9. (see playframework#11495). I think this question will be discussed again after release Play 2.9 and my personal opinion is Netty should be default.

[kromenak]

Hi, thanks for all the info! I had no idea about the licensing controversy around Akka HTTP. Based on that, plus our team's positive experience with Netty for the past several years, I think we will probably continue to use Netty for the time being.

[mkurz]

@kromenak Play 2.9 release candidate published: https://github.com/playframework/playframework/releases/tag/2.9.0-RC2 There you also find more detail about how we deal with the new Akka license

[kebeda]

@kromenak We have a play application that has been in production since 2013, and we've always followed the latest (switched to 2.8.19 in January, switched to 2.8.20 in July, and are running 2.9.0-M7 in development right now for deployment to prod later this year). We switched to Akka when play switched (2.6.?), but I would advise against it in your case...not only for all the drama in the links raised above, but because when we switched to Akka we were forced to bump up the fixed thread pool size to an insanely high number in order to get the concurrency we needed...we've since re-architected some things and have been able to achieve the same concurrency now with only an uncomfortably high fixed thread pool size, but the whole experience has left a really bad taste in the mouth. If you've been on Netty all this time you might as well stick with it until this whole Pekko/Scala 3 thing irons itself out, otherwise, you're bound to have to troubleshoot changes necessary for Akka 2.6 near term and Pekko later for the long term. I'd rather do that just once if I was forced to.

BTW, we are planning to switch back to netty for the 2.9.x release, FYI. I'm assuming Netty will continue to be an option going forward, no matter what.

[kromenak]

Hi there, thanks for providing your experience with switching to Akka HTTP!

Beyond the licensing drama, it is disconcerting to hear that you dealt with such scaling issues (sounds somewhat similar to the scaling problems mentioned in that discussion link from my original post). The last thing I want is to switch to Akka HTTP "just because" and encounter scaling issues in our next release! So yeah, we'll probably just continue to use Netty.

I do think the docs are a bit unclear about Play's preferred server backend. When a new version of Play says that Akka HTTP is now the default, it somewhat implies that it's somehow better than Netty or preferred to Netty, or that Netty support may be deprecated...but perhaps that's just me reading into it more than I should.

But if it's really just saying "we support multiple backends, pick whichever you'd like," then that makes sense too, and we'll probably stick with Netty until someone says an alternative is highly preferred.

[kebeda]

I can't speak for the motivations of the playframework team, but I believe they are working in good faith. To me Akka appears to have some superior features and architecture, particularly the focus on loose coupling, Actors and (if you need it) Streams. The question is...do you need it? And I think the question for the playframework team is, do the majority of playframework users need it? On a superficial level, cleaner architecture is usually better, but in Akka's case I think it would be better suited in a constellation of other Akka-based systems talking to each other. In our use case, most of what our play application talks to are other netty (Micronaut, Vert.x et. al.) microservices...in that scenario, the benefits don't seem so clear. You're only as strong as your weakest link and all that. That is not playframework's problem, however, it's ours. And I'm not asking them to solve it. I'm just glad the Netty option is still there so I can watch all this get resolved from the sidelines!

My reading of the documentation is that both backends were equally supported, but having been in the business long enough, I know that can't possibly be true. The one comment about the less-than-optimal state of the netty backend's unit tests is the tell, IMHO. I would expect an official pronouncement on this subject by the time 3.0 gets up to bat, if not sooner. Keep watching Pekko, there will probably be some tells there, as well.

[mkurz]

The one comment about the less-than-optimal state of the netty backend's unit tests is the tell, IMHO.

Which comment?

Netty and Akka HTTP (soon Pekko HTTP) are both tested equally. The Netty backend does receive the same amount of support. New backend server feature and fixes are applied to both backends, and there is no plan to drop Netty anytime, quite the opposite. E.g. when I implemented ping/pong support for WebSockets, I made sure both backends behave the same way and I put in extra effort for some edge cases where the Netty backend used to react differently than Akka HTTP, to make sure both backends can be replaced by one another without surprises.

Of course, both backends do have different performance characteristics and can be tweaked in different ways, but that's their nature.

@kromenak In your specific case, I recommend staying on Netty, especially since you've used it for quite a while already. There is no reason to switch to another server backend, and, as mentioned, Netty will continue to be supported in the future.