Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP/2 max local stream count exceeded #6603

Closed
prakashnandihal opened this issue Aug 12, 2021 · 24 comments · Fixed by #6639 or #6682
Closed

HTTP/2 max local stream count exceeded #6603

prakashnandihal opened this issue Aug 12, 2021 · 24 comments · Fixed by #6639 or #6682
Labels
Bug For general bugs on Jetty side

Comments

@prakashnandihal
Copy link

prakashnandihal commented Aug 12, 2021

Jetty version(s) 9.4.41.v20210516

Java version/vendor 1.8.0_271

OS type/version: Linux and Windows 10 machine

Description
HTTP/2 max local stream count exceeded

We were using jetty 9.4.35 version and we encountered this "HTTP/2 max local stream count 1 exceeded" error ,Simon suggested us to upgrade jetty version to 9.4.41 , so we did jetty jar upgrade to 9.4.41 version , but still we see the error .

Below is the code snippet we have written to create http2 client to post requests to destination host.

 HttpClient httpClient = null;
 HTTP2Client http2Client=null;
    try {
        http2Client = new HTTP2Client();
        http2Client.start();
       SslContextFactory sslContextFactory = new SslContextFactory.Client(true);
       final KeyStore sslKeyStore = loadKeystore(); //loadKeystore method loads keystire
      if (sslKeyStore != null) {
         sslContextFactory.setKeyStore(sslKeyStore);
       }
        httpClient = new HttpClient(new HttpClientTransportOverHTTP2(http2Client),
                sslContextFactory);
        httpClient.setMaxConnectionsPerDestination(100);
        httpClient.setMaxRequestsQueuedPerDestination(250000);
        httpClient.start();
        return httpClient;
    }
    catch (Exception exp) {
   }

How to reproduce?
Set the MaxConnectionsPerDestination value to 100 and try to post more than 10K requests.

@prakashnandihal prakashnandihal added the Bug For general bugs on Jetty side label Aug 12, 2021
@prakashnandihal
Copy link
Author

prakashnandihal commented Aug 12, 2021

Below is the Exception Stack trace

java.lang.IllegalStateException: Max local stream count 1 exceeded
	at org.eclipse.jetty.http2.HTTP2Session.createLocalStream(HTTP2Session.java:740) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.createLocalStream(HTTP2Session.java:2073) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.newLocalStream(HTTP2Session.java:2024) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$600(HTTP2Session.java:1429) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:577) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpSenderOverHTTP2.sendHeaders(HttpSenderOverHTTP2.java:110) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:212) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.send(HttpChannelOverHTTP2.java:98) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:128) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:233) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:87) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpDestinationOverHTTP2.send(HttpDestinationOverHTTP2.java:38) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:381) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:336) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:315) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpDestination.release(HttpDestination.java:443) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.release(HttpChannelOverHTTP2.java:106) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.exchangeTerminated(HttpChannelOverHTTP2.java:115) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:476) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:461) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:424) ~[jetty-client-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2.access$1400(HttpReceiverOverHTTP2.java:49) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2$ContentNotifier.process(HttpReceiverOverHTTP2.java:273) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2$ContentNotifier.offer(HttpReceiverOverHTTP2.java:234) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2$ContentNotifier.access$300(HttpReceiverOverHTTP2.java:214) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2.notifyContent(HttpReceiverOverHTTP2.java:211) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2.onData(HttpReceiverOverHTTP2.java:179) ~[http2-http-client-transport-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Stream.notifyData(HTTP2Stream.java:636) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Stream.onData(HTTP2Stream.java:387) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:306) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Session.onData(HTTP2Session.java:259) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onData(HTTP2Connection.java:390) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.parser.BodyParser.notifyData(BodyParser.java:108) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:150) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:145) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.parser.DataBodyParser.parse(DataBodyParser.java:111) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:198) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:127) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361) ~[http2-common-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) ~[jetty-io-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) ~[jetty-io-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) ~[jetty-io-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) ~[jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) [jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) [jetty-util-9.4.41.v20210516.jar:9.4.41.v20210516]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_271]

@sbordet
Copy link
Contributor

sbordet commented Aug 12, 2021

@prakashnandihal what's your configuration for maxConcurrentStreams on the server?

Do your request have a request body?

Do your request have a total timeout (i.e. request.timeout(...))?

Do you have other failures in your client-side logs, like TimeoutException, etc.?

@sbordet
Copy link
Contributor

sbordet commented Aug 12, 2021

Actually, can you attach a reproducible Maven project that shows the issue if it's that simple for you to reproduce?
I ask because we do the same kind of tests that you're doing, but we cannot reproduce the failure, so it must be some different configuration or something different that you do.

@prakashnandihal
Copy link
Author

prakashnandihal commented Aug 12, 2021

@sbordet Please find the answers inline.
what's your configuration for maxConcurrentStreams on the server?
Answer : Our Client application runs on tomcat server and makes request to destination host https://api.push.apple.com
using jetty http2 API, are you asking about maxConcurrentStreams configured on destination server(https://api.push.apple.com) ?

Do your request have a request body?
Answer : Yes We post Json string in request body.

Do your request have a total timeout (i.e. request.timeout(...))?
Answer : what you mean by do your request have a total timeout ?

Do your request have a total timeout (i.e. request.timeout(...))?
Answer : I will check and get back to you

can you attach a reproducible Maven project that shows the issue if it's that simple for you to reproduce?
Answer : I will check and get back to you

Additional Logs :
I enabled debug logs while posting requests , below logs are printed in my client server (tomcat)

(2021-08-12T21:27:18,322) DEBUG [-::HttpClient@29eff210-2287] [] [org.eclipse.jetty.client.HttpSender:387] Terminating request HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460
(2021-08-12T21:27:18,322) DEBUG [-::HttpClient@29eff210-2287] [] [org.eclipse.jetty.client.HttpSender:396] Response failure from request HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460 HttpExchange@2a4b7c00{req=HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded] res=HttpResponse[null 0 null]@e67647f[COMPLETED/java.lang.IllegalStateException: Max local stream count 1000 exceeded]}
(2021-08-12T21:27:20,406) DEBUG [-::HttpClient@29eff210-2287] [] [org.eclipse.jetty.client.HttpReceiver:545] Response abort HttpResponse[null 0 null]@e67647f HttpExchange@2a4b7c00{req=HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded] res=HttpResponse[null 0 null]@e67647f[COMPLETED/java.lang.IllegalStateException: Max local stream count 1000 exceeded]} on HttpChannelOverHTTP2@5ccedc68(exchange=HttpExchange@2a4b7c00{req=HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded] res=HttpResponse[null 0 null]@e67647f[COMPLETED/java.lang.IllegalStateException: Max local stream count 1000 exceeded]})[send=HttpSenderOverHTTP2@25134c89(req=FAILURE,snd=FAILED,failure=java.lang.IllegalStateException: Max local stream count 1000 exceeded),recv=HttpReceiverOverHTTP2@54c99f57(rsp=FAILURE,failure=java.lang.IllegalStateException: Max local stream count 1000 exceeded)]: {}
(2021-08-12T21:27:23,067) DEBUG [-::qtp1440282645-1922] [] [org.eclipse.jetty.client.HttpExchange:228] Failed HttpExchange@2a4b7c00{req=HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded] res=HttpResponse[null 0 null]@e67647f[COMPLETED/java.lang.IllegalStateException: Max local stream count 1000 exceeded]}: req=false/rsp=false {}
(2021-08-12T21:27:23,211) DEBUG [-::HttpClient@29eff210-2287] [] [org.eclipse.jetty.client.HttpExchange:210] Terminated response for HttpExchange@2a4b7c00{req=HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded] res=HttpResponse[null 0 null]@e67647f[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded]}, result: Result[HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460 > HttpResponse[null 0 null]@e67647f] java.lang.IllegalStateException: Max local stream count 1000 exceeded
(2021-08-12T21:27:23,884) DEBUG [-::HttpClient@29eff210-2287] [] [org.eclipse.jetty.client.HttpChannel:100] HttpExchange@2a4b7c00{req=HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded] res=HttpResponse[null 0 null]@e67647f[TERMINATED/java.lang.IllegalStateException: Max local stream count 1000 exceeded]} disassociated true from HttpChannelOverHTTP2@5ccedc68(exchange=null)[send=HttpSenderOverHTTP2@25134c89(req=FAILURE,snd=FAILED,failure=java.lang.IllegalStateException: Max local stream count 1000 exceeded),recv=HttpReceiverOverHTTP2@54c99f57(rsp=FAILURE,failure=java.lang.IllegalStateException: Max local stream count 1000 exceeded)]
(2021-08-12T21:27:32,250) DEBUG [-::HttpClient@29eff210-2287] [] [org.eclipse.jetty.client.HttpReceiver:478] Request/Response failed: Result[HttpRequest[POST /3/device/3513a176f516b442b84f4c8c3471f9869356d9f1cd957690e117a63b6270 HTTP/2.0]@26dd460 > HttpResponse[null 0 null]@e67647f] java.lang.IllegalStateException: Max local stream count 1000 exceeded

@joakime
Copy link
Contributor

joakime commented Aug 12, 2021

From https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/

"APNs allows multiple concurrent streams for each connection, but don’t assume a specific number of streams. The exact number varies based on server load and whether you use a provider certificate or an authentication token. For example, when using an authentication token, APNs allows only one stream until you post a request with a valid authentication token. APNs ignores HTTP/2 PRIORITY frames, so don’t send them on your streams."

@joakime joakime changed the title HTTP/2 max local stream count exceeded Http2 Client - HTTP/2 max local stream count exceeded when connected to Apple APN Aug 12, 2021
@sbordet sbordet changed the title Http2 Client - HTTP/2 max local stream count exceeded when connected to Apple APN HTTP/2 max local stream count exceeded Aug 12, 2021
@sbordet
Copy link
Contributor

sbordet commented Aug 12, 2021

@joakime I changed the title back to original. To what server the client is connecting to is irrelevant, this is a client issue.

@gregw
Copy link
Contributor

gregw commented Aug 12, 2021

So if I'm reading between the lines of what @joakime and @sbordet are saying.... the issue might be that the APN server is setting the max streams per connection to 1 (which is the limit that we see in the exception) in such a way that it causes a race in our client connection pool. The client should never see this exception because when the max streams is exceeded a new connection is created.... but there is a limit of 100 connections per destination. So perhaps the issue is that once those 100 connections are used up, a request is getting past the point where it should queue waiting for a connection to become available and actually attempts to create a stream on an existing connection - this creation races with the server setting the limit to 1, so the limit is exceeded.

So maybe we should have a similar test where the server always sets the stream limit to 1 whilst the client is receiving many requests for a connection limited destination?

@prakashnandihal
Copy link
Author

@sbordet : I have attached maven project for your reference.
https://github.com/prakashnandihal/jetty-http2client-repo.git

As per apple developer documentation , APNS supports multiple concurrent streams and number of concurrent streams varies depending upon the server load.

https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/

Can you please explain what is wrong with client?

@sbordet
Copy link
Contributor

sbordet commented Aug 17, 2021

@prakashnandihal the token is already expired, unfortunately, so I just get 403.

Can you please run your test project with the system property -Dorg.eclipse.jetty.http2.LEVEL=DEBUG, then gzip and attach the resulting log file to this issue?

@prakashnandihal

This comment has been minimized.

@sbordet
Copy link
Contributor

sbordet commented Aug 17, 2021

@prakashnandihal trying.

@prakashnandihal
Copy link
Author

@sbordet
In my code, i am making post to dummy device tokens , we get below json response from APNS
{"reason":"BadDeviceToken"} and Status code:400
, BadDeviceToken error reason is expected as per APNS documentation for the dummy device tokens. And for some requests i see below error :
java.lang.IllegalStateException: Max local stream count 1 exceeded

sbordet added a commit that referenced this issue Aug 18, 2021
Made MAX_CONCURRENT_STREAMS setting work on a per-connection basis.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
@sbordet sbordet linked a pull request Aug 18, 2021 that will close this issue
@sbordet
Copy link
Contributor

sbordet commented Aug 18, 2021

@prakashnandihal I think we have found the problem.

It should be fixed in this branch: jetty-9.4.x-6603-max-concurrent-streams-exceeded.
Are you able to try it out to confirm the fix?

@prakashnandihal
Copy link
Author

prakashnandihal commented Aug 18, 2021

@sbordet yes i will try

After cloning jetty-9.4.x-6603-max-concurrent-streams-exceeded branch, i hope below steps are correct to test the fix, please confirm
1)go to directory jetty-9.4.x-6603-max-concurrent-streams-exceeded and run mvn clean install
2)use this jetty 9.4.44-SNAPSHOT version in my maven project's pom.xml
3)Build and Run the application

@joakime
Copy link
Contributor

joakime commented Aug 18, 2021

@prakashnandihal

These are the commands ...

$ git clone https://github.com/eclipse/jetty.project.git
$ cd jetty.project
$ git checkout jetty-9.4.x-6603-max-concurrent-streams-exceeded
$ mvn clean install -DskipTests

Then use 9.4.44-SNAPSHOT in your project.

@prakashnandihal
Copy link
Author

@joakime : Thanks for providing info.

@sbordet : Issue is Fixed. I tested fix from jetty-9.4.x-6603-max-concurrent-streams-exceeded branch, i did not see this error "max local stream count exceeded".

When will this fix publicly available ?

When i set High value like 500 or 1000 to maxConnectionsPerDestination, i got below intermittent errors
1)java.nio.channels.ClosedChannelException
2)java.net.SocketException: Permission denied: no further information
3)java.net.SocketException: No buffer space available (maximum connections reached?): connect.

I have below couple of questions on Jetty Http2 API. Please provide your inputs

Currently we are using jetty Http2 API for sending push notifications in production and Now
We have new requirement to send 80,000 Requests per second, we want to know what is
maximum throughput we can get After tweeking all configurations in http2 client from one instance of application for one destination?

And also please recommend us what are all configuration parameters we can tweak to enhance the throughput of http2client.

And also I want to Understand more on MaxRequestsQueuedPerDestination and MaxConnectionsPerDestination properties of HttpClient object.

  1. MaxRequestsQueuedPerDestination : does jetty creates one queue per destination ? or one queue per connection+destination combination?

Example:
let us say If I am creating 10 httpclient objects using same SSL credentials
for posting requests to Same Destination and We set MaxRequestsQueuedPerDestination
value to 100K for each httpclient object. does jetty maintains queue of size 100K for each httpclient to store the requests?
or Does it creates only queue of size 100K for the destination ?

2)MaxConnectionsPerDestination:
let us say If I am creating 10 httpclient objects using same SSL credentials
for posting requests to Same Destination and we set MaxConnectionsPerDestination value 100 to each httpclient object,
does it mean jetty allocates (10 httpclients * 100 ==1000) connections to Destination?

@joakime
Copy link
Contributor

joakime commented Aug 19, 2021

let us say If I am creating 10 httpclient objects using same SSL credentials

This is not recommended.
Create only 1 HttpClient instance, ever.
The only reason to have more than 1 HttpClient is if you simultaneously want to stay on Jetty 9 and support both HTTP/1.1 and HTTP/2 at the same time (then you'll have at most 2 HttpClient instances, one configured for HTTP/1.1 exclusively and one configured for HTTP/2 exclusively).
If you move to Jetty 10, then even this is moot, as under Jetty 10, it can support HTTP/1.1 and HTTP/2 on the same HttpClient instance.

@prakashnandihal
Copy link
Author

let us say If I am creating 10 httpclient objects using same SSL credentials

This is not recommended.
Create only 1 HttpClient instance, ever.
The only reason to have more than 1 HttpClient is if you simultaneously want to stay on Jetty 9 and support both HTTP/1.1 and HTTP/2 at the same time (then you'll have at most 2 HttpClient instances, one configured for HTTP/1.1 exclusively and one configured for HTTP/2 exclusively).
If you move to Jetty 10, then even this is moot, as under Jetty 10, it can support HTTP/1.1 and HTTP/2 on the same HttpClient instance.

@joakime
Let me explain our use case, we have integrated with APNS to send push notification, We can send notifications to any number of ios apps using APNS.
Let say we want to send push notification 5 different Apps using single destination(APNS) , Every app will have unique credentials .
When we are sending push notifications to all the five app using Jetty http2 library, we have to create 5 instances of HttpClient class because each app has unique SSL credentials.
In this case if I have set MaxConnectionsPerDestination value as
100 and MaxRequestsQueuedPerDestination value as 5000 for each HttpClient object,
does Jetty creates total 500 Connection for APNS destination or 100 for Destination?
does jetty creates 5 queues, each queue with size 5000 to hold requests?

@joakime
Copy link
Contributor

joakime commented Aug 19, 2021

You still don't need multiple HttpClient instances for that.

You should be able to use the Request tagging for those 5 applications. (see Request.tag(Object))
Each tag would represent the individual application.
Which will isolate those applications to those specific connections in the ConnectionPool with the same tag.

@sbordet sbordet added this to To do in Jetty 9.4.44 FROZEN via automation Aug 19, 2021
@sbordet
Copy link
Contributor

sbordet commented Aug 19, 2021

Issue is Fixed.

Thanks for the feedback!

When will this fix publicly available ?

With Jetty 9.4.44, the date is not set yet.

i got below intermittent errors

All of them seem caused by an underconfigured machine (a machine that is too small or configured too small for the job).

We have new requirement to send 80,000 Requests per second

We cannot tell you the max throughput, you have to measure it.
It depends on the size of the requests, the response, the network speed, the OS, the JVM version, and probably a million other things that I can't remember on top of my head :)

With multiple machines will be easier, provided the network is not a bottleneck. At 80k requests/s you saturate a 1 Gib network with about 1000000000/8/80000=~1500 bytes per message.

MaxRequestsQueuedPerDestination : does jetty creates one queue per destination ? or one queue per connection+destination combination?

One queue per destination.
The queue is not pre-allocated, but it is bounded. If you never queue requests because the connections are enough to always send them to the server, then the queue will occupy little space.

we set MaxConnectionsPerDestination value 100 to each httpclient object,
does it mean jetty allocates (10 httpclients * 100 ==1000) connections to Destination?

Correct.

we have to create 5 instances of HttpClient class because each app has unique SSL credentials.

This is currently simpler than using the tag() mechanism, although we should really improve the tag() mechanism to support different SslContextFactory.Client instances.

With 5 HttpClients each with MaxConnectionsPerDestination=100, there will be at most 5000 connections to the server. However, connections are only created if needed. If 1 connection per HttpClient is enough to sustain the load, only 5 connections will be opened to the server.

sbordet added a commit that referenced this issue Aug 19, 2021
Updates after review.
Updated the maxMultiplex mechanism to always work on Pool.Entry, rather than on Pool.

Updated Pool javadocs.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
sbordet added a commit that referenced this issue Aug 20, 2021
Fixed javadocs.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
@prakashnandihal
Copy link
Author

@sbordet Thanks For Your Info.
can you please tell us tentative date for 9.4.44 release?

@sbordet
Copy link
Contributor

sbordet commented Aug 23, 2021

@prakashnandihal we are still discussing what is the best fix for this issue, so we don't have a date yet.

sbordet added a commit that referenced this issue Aug 27, 2021
Made MAX_CONCURRENT_STREAMS setting work on a per-connection basis.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
sbordet added a commit that referenced this issue Aug 27, 2021
Updates after review.
Updated the maxMultiplex mechanism to always work on Pool.Entry, rather than on Pool.

Updated Pool javadocs.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
sbordet added a commit that referenced this issue Aug 27, 2021
Fixed javadocs.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
sbordet added a commit that referenced this issue Aug 27, 2021
Incorporated changes from #6648.

Now the maxMultiple value is pulled from its primary value,
i.e. HTTP2Session.maxLocalStreams, rather than being set
in multiple places.

Deprecated usages of maxMultiplex and maxUsageCount.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
sbordet added a commit that referenced this issue Aug 30, 2021
Added comment as suggested in review.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
@prakashnandihal
Copy link
Author

@sbordet : We have to release fix for this bug to production Pod, Please notify us the once the release date of 9.4.44 is finalized. so that We will plan release of this fix to production pod.

sbordet added a commit that referenced this issue Aug 30, 2021
Made newEntry() private to avoid creation of more than 2 Entry subclasses.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
sbordet added a commit that referenced this issue Aug 30, 2021
Made MAX_CONCURRENT_STREAMS setting work on a per-connection basis.
Updated Pool javadocs.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Co-authored-by: Greg Wilkins <gregw@webtide.com>
sbordet added a commit that referenced this issue Aug 30, 2021
Made MAX_CONCURRENT_STREAMS setting work on a per-connection basis.
Updated Pool javadocs.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Co-authored-by: Greg Wilkins <gregw@webtide.com>
(cherry picked from commit 525fcb3)
@sbordet sbordet closed this as completed in e2690cc Sep 1, 2021
Jetty 9.4.44 FROZEN automation moved this from To do to Done Sep 1, 2021
@sbordet sbordet linked a pull request Sep 1, 2021 that will close this issue
@prakashnandihal
Copy link
Author

@sbordet what is the release date of 9.4.44?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For general bugs on Jetty side
Projects
No open projects
4 participants