HttpServerRequest supports query parameters. #2696

jchenga · 2023-02-15T06:38:56Z

This PR is for issue #68. I have done an implementation to add support for query parameters in HttpServerRequest using Netty's QueryStringDecoder.

I think more needs to be done to fully flesh out the feature, such as injecting a querystring decoder instead of newing one up. @violetagg and @pderop any suggestion is appreciated.

pderop · 2023-02-15T07:52:51Z

@jchenga ,

thanks for this PR ! I will look into it soon today.

pderop · 2023-02-15T13:18:29Z

@violetagg ,

I have set the target milestone to 1.1.4 since it's an enhancement. Let me know if we should backport this to 1.0.x branch ?
thanks

pderop

@jchenga ,

So, I have taken a look, can you please check the following reviews ?
In addition, I have the following comments:

can you rebase your PR on top of the latest main branch, because for japicmp, you need to update the reactor-netty-http/build.gradle, which has been updated yesterday when we did the release ? thanks.
japicmp:
in order to avoid the japicmp error, can you please update the reactor-netty-http/build.gradle, and exclude the new queryParams()method, something like this line 244:

	methodExcludes = [
			'reactor.netty.http.server.HttpServerRequest#queryParams()'
	]

(in the latest branch, the methodExcludes is now empty because we have made the release yesterday, so just add your new method).

I'm not sure if there is more works to be done, such as injecting a querystring decoder ? Indeed the decoder seems to be immutable, and contains only few objects, so why would we have to inject a reusable decoder ? But maybe I missed your point ?

thank you !

pderop · 2023-02-15T13:22:30Z

reactor-netty-http/src/main/java/reactor/netty/http/server/HttpServerOperations.java

+	private Map<String, List<String>> parseQueryParams(HttpRequest request) {
+		QueryStringDecoder decoder = new QueryStringDecoder(request.uri());
+		return decoder.parameters();
+	}


in order to avoid mixing public and private methods, what do you think about moving the private parseQueryParams method just before the static inner classes that are declared in the end of the class ?

pderop · 2023-02-15T13:28:48Z

reactor-netty-http/src/main/java/reactor/netty/http/server/HttpServerRequest.java

+	 * @return query parameters {@link Map&lt;String,List&lt;String&gt;&gt;}
+	 */


Suggested change

* @return query parameters {@link Map<String,List<String>>}

*/

* @return query parameters {@link Map}

*/

To avoid some compilation warnings, I think it is safe to just remove the generic types in the javadoc link.

reactor-netty-http/src/test/java/reactor/netty/http/server/HttpServerTests.java

reactor-netty-http/src/main/java/reactor/netty/http/server/HttpServerRequest.java

violetagg · 2023-02-16T09:03:59Z

@jchenga Can we limit this feature only to route functionality similar to how it is done for parameters resolution (see reactor.netty.http.server.DefaultHttpServerRoutes.HttpRouteHandler and reactor.netty.http.server.HttpPredicate)? Frameworks typically use handle and they may want to use something else for query resolution.

jchenga · 2023-02-16T17:43:18Z

@violetagg thanks! i can look into limiting the feature in route. out of curiosity, why do we want to limit this to route only? wouldn't we want to make query params widely available?

violetagg · 2023-02-16T18:46:10Z

@jchenga Take a look at the parameters resolver. You can provide a resolver so that if you use handle we can use this resolver to give you want you want. The point is that frameworks that use handle may not want to use the Netty's query resolver but something else (e.g. Spring Framework).

reactor-netty/reactor-netty-http/src/main/java/reactor/netty/http/server/HttpServerRequest.java

Line 70 in 906de4b

    
           HttpServerRequest paramsResolver(Function<? super String, Map<String, String>> paramsResolver);

I also need to take a look at the Netty's query resolver. It seems it resolves everything? if that's true we will have double parsing of the uri or not?

reactor-netty/reactor-netty-http/src/main/java/reactor/netty/http/server/HttpServerOperations.java

Line 194 in 906de4b

this.path = resolvePath(nettyRequest.uri());

violetagg · 2023-02-16T18:56:04Z

@jchenga I see now...

Can we discuss with Netty maintainers to make decodeParams public similar to decodeComponent as basically we need just this utility or it is not the case?

violetagg · 2023-02-16T19:01:04Z

I mean I don't see why we want to create this object every time, basically QueryStringDecoder holds mostly the same information as our HttpServerOperations, isn't it?

jchenga · 2023-02-21T05:49:51Z

There is definitely no need to create an QueryStringDecoder object every time parseQueryParams method is called. I tried to eliminate the need to recreate objects by including the null check on the map.

reactor-netty/reactor-netty-http/src/main/java/reactor/netty/http/server/HttpServerOperations.java

Lines 467 to 469 in 1e693a9

    
           if (queryParamsMap == null) { 
        
           	queryParamsMap = Collections.unmodifiableMap(parseQueryParams(this.nettyRequest)); 
        
           }

That does seem a bit hacky and I see your point about having to create an object every time we need to parse query params. I can reach out to Netty's maintainers to see if they are willing to make decodeParams public.

another option is to bring in decodeParams codes from Netty into HttpOperations. Since decodeParams is trivial relative to other logics, perhaps we can just bring in the function into this project? Or maybe reusing the similar function in spring is an option too?

violetagg · 2023-02-22T09:17:12Z

another option is to bring in decodeParams codes from Netty into HttpOperations. Since decodeParams is trivial relative to other logics, perhaps we can just bring in the function into this project?

My opinion is that forking/copying comes with a drawback that we need always to check whether there are some changes to the original code.

So my proposal is to ask Netty maintainers. If this suggestion is not ok from their point of view, we can discuss the other two options. Wdyt?

jchenga · 2023-02-22T18:07:41Z

another option is to bring in decodeParams codes from Netty into HttpOperations. Since decodeParams is trivial relative to other logics, perhaps we can just bring in the function into this project?

My opinion is that forking/copying comes with a drawback that we need always to check whether there are some changes to the original code.

So my proposal is to ask Netty maintainers. If this suggestion is not ok from their point of view, we can discuss the other two options. Wdyt?

Sounds good to me. I'll create a request over at the Netty's repo for this.

jchenga · 2023-02-22T18:47:27Z

I have created a request in netty netty/netty#13239 to make decodeParams public.

jchenga · 2023-03-15T16:44:53Z

@violetagg it looks like Netty folks are not keen on exposing decodeParams as public API. Let's go with bringing the codes into this project's codebase? what do you think?

violetagg · 2023-03-22T10:02:07Z

@violetagg it looks like Netty folks are not keen on exposing decodeParams as public API. Let's go with bringing the codes into this project's codebase? what do you think?

I will need to take a closer look at this one ...

jchenga · 2023-03-26T04:30:00Z

@violetagg it looks like Netty folks are not keen on exposing decodeParams as public API. Let's go with bringing the codes into this project's codebase? what do you think?

I will need to take a closer look at this one ...

Let me know once you know what you would like to do and I'll jump in to help.

violetagg · 2023-03-28T07:01:32Z

@violetagg it looks like Netty folks are not keen on exposing decodeParams as public API. Let's go with bringing the codes into this project's codebase? what do you think?

@jchenga Let's go with bringing this code to Reactor Netty

jchenga · 2023-04-01T17:53:27Z

sure thing. let me get started on that.

jchenga · 2023-04-04T18:23:44Z

@violetagg here is what i am thinking. I'll bring in the QueryStringDecoder from Netty, remove logics not related to decoding parameters, and add two public methods below:

public static Map<String, List<String>> decodeParams(final String uri) {

		return decodeParams(uri, HttpConstants.DEFAULT_CHARSET,
				DEFAULT_MAX_PARAMS, true);
}

public static Map<String, List<String>> decodeParams(final String uri, final Charset charset, final int maxParams, final boolean semicolonIsNormalChar) {
		checkNotNull(uri, "uri");
		checkNotNull(charset, "charset");
		checkPositive(maxParams, "maxParams");

		int from = findPathEndIndex(uri);
		return decodeParams(uri, from, charset,
				maxParams, semicolonIsNormalChar);
	}

checkNotNull and checkPositive are Netty utility methods. Do we have equivalent methods? I'll also bring in the unit test and modify it to suit our needs.

violetagg · 2023-04-04T18:44:25Z

checkNotNull and checkPositive are Netty utility methods. Do we have equivalent methods? I'll also bring in the unit test and modify it to suit our needs.

Something like this

reactor-netty/reactor-netty-core/src/main/java/reactor/netty/transport/NameResolverProvider.java

Line 735 in 10c127a

this.queryTimeout = Objects.requireNonNull(queryTimeout, "queryTimeout");

and this

reactor-netty/reactor-netty-core/src/main/java/reactor/netty/transport/NameResolverProvider.java

Lines 717 to 719 in 10c127a

    
           if (maxQueriesPerResolve < 1) { 
        
           	throw new IllegalArgumentException("maxQueriesPerResolve must be positive"); 
        
           }

jchenga · 2023-08-27T03:35:27Z

@violetagg I've finally go time to pick this up again. the PR failed the API compatibility check because one new method was added to the HttpServerRequest interface. What should I do to resolve this failure?

violetagg · 2023-08-30T11:11:47Z

@violetagg I've finally go time to pick this up again. the PR failed the API compatibility check because one new method was added to the HttpServerRequest interface. What should I do to resolve this failure?

Can you try adding this?

diff --git a/reactor-netty-http/build.gradle b/reactor-netty-http/build.gradle
index 02b717a63..3272339ed 100644
--- a/reactor-netty-http/build.gradle
+++ b/reactor-netty-http/build.gradle
@@ -250,6 +250,7 @@ task japicmp(type: JapicmpTask) {
 
 	compatibilityChangeExcludes = [ "METHOD_NEW_DEFAULT" ]
 	methodExcludes = [
+			'reactor.netty.http.server.HttpServerRequest#queryParams()'
 	]
 }

jchenga · 2023-09-04T17:43:38Z

reactor-netty-http/src/main/java/reactor/netty/http/QueryStringDecoder.java

@@ -0,0 +1,203 @@
+/*
+ * Copyright (c) 2012-2023 VMware, Inc. or its affiliates, All Rights Reserved.


since the codes came from Netty, should the license statement be replaced with the one from Netty?

@violetagg could you have a look and see if the PR is good to go?

jchenga · 2023-09-04T17:47:32Z

@violetagg I've finally go time to pick this up again. the PR failed the API compatibility check because one new method was added to the HttpServerRequest interface. What should I do to resolve this failure?

Can you try adding this?
diff --git a/reactor-netty-http/build.gradle b/reactor-netty-http/build.gradle
index 02b717a63..3272339ed 100644
--- a/reactor-netty-http/build.gradle
+++ b/reactor-netty-http/build.gradle
@@ -250,6 +250,7 @@ task japicmp(type: JapicmpTask) {
 
 	compatibilityChangeExcludes = [ "METHOD_NEW_DEFAULT" ]
 	methodExcludes = [
+			'reactor.netty.http.server.HttpServerRequest#queryParams()'
 	]
 }

done. that resolved the issue.

violetagg · 2023-10-05T05:52:46Z

@jchenga Can you please rebase?

jchenga · 2023-10-16T01:13:16Z

@violetagg I have rebased the branch. the windows 2019, nio build failed, but it does not seem to be related to this PR

jchenga · 2023-11-08T20:57:08Z

Hi @violetagg I rebased the feature branch on main and some checks failed. Looking at the errors, I don't think the issue is related to the change in the PR. the failures seem to OS specific as "Check Matrix / build (ubuntu-20.04, native) (pull_request) " and "Check Matrix / build (ubuntu-20.04, nio) (pull_request) " were successful.

I am stumped. would you be able to help?

violetagg · 2023-11-09T06:15:27Z

Hi @violetagg I rebased the feature branch on main and some checks failed. Looking at the errors, I don't think the issue is related to the change in the PR. the failures seem to OS specific as "Check Matrix / build (ubuntu-20.04, native) (pull_request) " and "Check Matrix / build (ubuntu-20.04, nio) (pull_request) " were successful.

I am stumped. would you be able to help?

I ran again the tests and now they are ok

jchenga · 2023-11-09T17:51:37Z

Hi @violetagg I rebased the feature branch on main and some checks failed. Looking at the errors, I don't think the issue is related to the change in the PR. the failures seem to OS specific as "Check Matrix / build (ubuntu-20.04, native) (pull_request) " and "Check Matrix / build (ubuntu-20.04, nio) (pull_request) " were successful.
I am stumped. would you be able to help?

I ran again the tests and now they are ok

Thank you for the help. This PR is ready to be merged if everything looks ok.

pderop self-assigned this Feb 15, 2023

pderop added the type/enhancement A general enhancement label Feb 15, 2023

pderop added this to the 1.1.4 milestone Feb 15, 2023

pderop reviewed Feb 15, 2023

View reviewed changes

violetagg modified the milestones: 1.1.4, 1.1.x Backlog Mar 2, 2023

violetagg modified the milestones: 1.1.x Backlog, 1.1.7 Apr 7, 2023

violetagg modified the milestones: 1.1.7, 1.1.x Backlog May 2, 2023

violetagg unassigned pderop Jun 22, 2023

violetagg marked this pull request as draft June 29, 2023 05:53

jchenga marked this pull request as ready for review August 27, 2023 03:10

jchenga commented Sep 4, 2023

View reviewed changes

jchenga force-pushed the 68-query-parameters-in-request branch 3 times, most recently from 6f73ec8 to ac78a0b Compare October 16, 2023 00:06

HttpServerRequest supports query parameters.

80c2ee8

jchenga force-pushed the 68-query-parameters-in-request branch from ac78a0b to 80c2ee8 Compare November 8, 2023 19:10

violetagg modified the milestones: 1.1.x Backlog, 1.2.x Backlog Dec 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HttpServerRequest supports query parameters. #2696

HttpServerRequest supports query parameters. #2696

jchenga commented Feb 15, 2023

pderop commented Feb 15, 2023

pderop commented Feb 15, 2023 •

edited

pderop left a comment

pderop Feb 15, 2023

pderop Feb 15, 2023

violetagg commented Feb 16, 2023

jchenga commented Feb 16, 2023 •

edited

violetagg commented Feb 16, 2023

violetagg commented Feb 16, 2023

violetagg commented Feb 16, 2023

jchenga commented Feb 21, 2023 •

edited

violetagg commented Feb 22, 2023

jchenga commented Feb 22, 2023

jchenga commented Feb 22, 2023

jchenga commented Mar 15, 2023

violetagg commented Mar 22, 2023

jchenga commented Mar 26, 2023

violetagg commented Mar 28, 2023

jchenga commented Apr 1, 2023

jchenga commented Apr 4, 2023 •

edited

violetagg commented Apr 4, 2023

jchenga commented Aug 27, 2023

violetagg commented Aug 30, 2023

jchenga Sep 4, 2023

jchenga Oct 5, 2023

jchenga commented Sep 4, 2023

violetagg commented Oct 5, 2023

jchenga commented Oct 16, 2023

jchenga commented Nov 8, 2023

violetagg commented Nov 9, 2023

jchenga commented Nov 9, 2023

		* @return query parameters {@link Map<String,List<String>>}
		*/

		@@ -0,0 +1,203 @@
		/*
		* Copyright (c) 2012-2023 VMware, Inc. or its affiliates, All Rights Reserved.

HttpServerRequest supports query parameters. #2696

Are you sure you want to change the base?

HttpServerRequest supports query parameters. #2696

Conversation

jchenga commented Feb 15, 2023

pderop commented Feb 15, 2023

pderop commented Feb 15, 2023 • edited

pderop left a comment

Choose a reason for hiding this comment

pderop Feb 15, 2023

Choose a reason for hiding this comment

pderop Feb 15, 2023

Choose a reason for hiding this comment

violetagg commented Feb 16, 2023

jchenga commented Feb 16, 2023 • edited

violetagg commented Feb 16, 2023

violetagg commented Feb 16, 2023

violetagg commented Feb 16, 2023

jchenga commented Feb 21, 2023 • edited

violetagg commented Feb 22, 2023

jchenga commented Feb 22, 2023

jchenga commented Feb 22, 2023

jchenga commented Mar 15, 2023

violetagg commented Mar 22, 2023

jchenga commented Mar 26, 2023

violetagg commented Mar 28, 2023

jchenga commented Apr 1, 2023

jchenga commented Apr 4, 2023 • edited

violetagg commented Apr 4, 2023

jchenga commented Aug 27, 2023

violetagg commented Aug 30, 2023

jchenga Sep 4, 2023

Choose a reason for hiding this comment

jchenga Oct 5, 2023

Choose a reason for hiding this comment

jchenga commented Sep 4, 2023

violetagg commented Oct 5, 2023

jchenga commented Oct 16, 2023

jchenga commented Nov 8, 2023

violetagg commented Nov 9, 2023

jchenga commented Nov 9, 2023

pderop commented Feb 15, 2023 •

edited

jchenga commented Feb 16, 2023 •

edited

jchenga commented Feb 21, 2023 •

edited

jchenga commented Apr 4, 2023 •

edited