HttpClient is encoding ampersands and equal signs in my get request

[Jonathan-Zollinger]

Short Version

after reading the httpclient logs, I'm seeing that some characters are being url-encoded. I dont want that to happen, is that possible?

Long Version

the url I'm calling is split into <base url>+<path>+<apitoken>+<query> + <channel>. The query is simply <tcins> then <store info>. there are = and & peppered through those, but these characters are encoded to %26 and %3D inconsistently through the url. for instance you'll note that calling the api key is fine to use ?key=..., but immediately afterwards, the &tcins=82691535... is encoded to %26tcins%3D82691535....

the get method is:

   @Get("/product_summary_with_fulfillment_v1?key=${redsky-api.token}&{tcins}&{targetStore}&channel=WEB")
    HttpResponse<List<?>> fetchProducts(String tcins, String targetStore);

Log excerpt

11:58:04.258 [default-nioEventLoopGroup-1-1] DEBUG io.micronaut.http.client.netty.DefaultHttpClient - Sending HTTP GET to  https://redsky.target.com/redsky_aggregations/v1/web/product_summary_with_fulfillment_v1?key=9f36aeafbe60771e321a7cc95a78140772ab3e96&%26tcins%3D82691535&store_id%3D1750%26zip%3D84014-1752%26state%3DUT%26latitude%3D40.91825%26longitude%3D-111.887%26scheduled_delivery_store_id%3D1750%26required_store_id%3D1750%26has_required_store_id%3Dtrue&channel=WEB
11:58:04.265 [default-nioEventLoopGroup-1-1] TRACE io.micronaut.http.client.netty.DefaultHttpClient - User-Agent: Micronaut HTTP Client
11:58:04.265 [default-nioEventLoopGroup-1-1] TRACE io.micronaut.http.client.netty.DefaultHttpClient - Accept: application/vnd.github.v3+json, application/json
11:58:04.266 [default-nioEventLoopGroup-1-1] TRACE io.micronaut.http.client.netty.DefaultHttpClient - host: redsky.target.com

for comparison:

Bad Url	Proper Url
&store_id%3D1750 %26zip%3D84014-1752 %26state%3DUT %26latitude%3D40.91825 %26longitude%3D-111.887 %26required_store_id%3D1750 %26has_required_store_id%3Dtrue	&store_id=1750 &zip=84010 &state=UT &latitude=40.894523 &longitude=-111.872400 &required_store_id=1750 &required_store_id=1750

Controller

package com.github.peanutbutter.unicorn.tarvester;

import com.github.peanutbutter.unicorn.tarvester.model.TargetStore;
import com.github.peanutbutter.unicorn.tarvester.model.TcinList;
import io.micronaut.core.async.annotation.SingleResult;
import io.micronaut.http.HttpResponse;
import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;

import java.util.List;

@Controller("products")
public class TarvesterController {

    final TarvesterClient tarvesterClient;

    public TarvesterController(TarvesterClient tarvesterClient) {
        this.tarvesterClient = tarvesterClient;
    }

    @Get("by-tcin")
    @SingleResult
    HttpResponse<List<?>> fetchProducts(TcinList tcinlist, TargetStore targetStore) {
        return tarvesterClient.fetchProducts(tcinlist.toString(), targetStore.toString());
    }
}

Client

package com.github.peanutbutter.unicorn.tarvester;

import com.github.peanutbutter.unicorn.tarvester.model.Product;
import io.micronaut.http.HttpResponse;
import io.micronaut.http.annotation.Get;
import io.micronaut.http.annotation.Header;
import io.micronaut.http.client.annotation.Client;

import java.util.List;

import static io.micronaut.http.HttpHeaders.ACCEPT;
import static io.micronaut.http.HttpHeaders.USER_AGENT;

@Client(id = "redsky-api")
@Header(name = USER_AGENT, value = "Micronaut HTTP Client")
@Header(name = ACCEPT, value = "application/vnd.github.v3+json, application/json")
public interface TarvesterClient {

    @Get("/product_summary_with_fulfillment_v1?key=${redsky-api.token}&{targetStore}&{tcins}&channel=WEB")
    HttpResponse<List<?>> fetchProducts(String tcins, String targetStore);
}

ToString methods

    //tcins excerpt
    @Override
    public String toString() {
        return "&tcins=" + Arrays.stream(id)
                .mapToObj(String::valueOf)
                .collect(Collectors.joining("%2C"));
    }
...
    // TargetStore excerpt
    @Override
    public String toString() {
        return String.join("&", new String[]{
                "store_id=" + locationId,
                "zip=" + postalCode,
                "state=" + region,
                "latitude=" + latitude,
                "longitude=" + longitude,
                "scheduled_delivery_store_id=" + locationId,
                "required_store_id=" + locationId,
                "has_required_store_id=true"
        });
    }

application.properties

micronaut.application.name=Tarvester
micronaut.application.default-charset=UTF-8
micronaut.http.services.redsky-api.url=https://redsky.target.com/
micronaut.http.services.redsky-api.path=redsky_aggregations/v1/web
redsky-api.token=9f36aeafbe60771e321a7cc95a78140772ab3e96

Things I've tried

• I've tried changing the controller and get annotations (separately) to include produces = MediaType.PLAINTEXT, but neither of those effected the url encoding issue.
• I am swapping out where the java objects are converted to strings (which is where these characters are introduced) to be in the client or the controller, but i'm not hopeful this will work as the Get annotation includes the api key retrieval as well as the string interpolation for the java object string values.
• I can't find any documentation on some kind of parameter for the httpclient to not encode the url
• I read in the configuration reference that you can define the default application charset, so I changed that to UTF-8 to no avail

if you want to try this yourself, I've put this on its own branch on my repo:

Part of me wonders if this is a bug since this isn't consisten behavoir across the URL?

[graemerocher]

one thing to try, we follow the URI templates spec so you may need a + in front of the variables to avoid escaping. The server and the client both use the same uri template implementation. See https://docs.micronaut.io/latest/guide/#routing section URI Reserved Character Matching (which seems to have a rendering problem in the docs)

[Jonathan-Zollinger]

I appreciate the warning - I'll read into how I might mitigate that.

[Jonathan-Zollinger]

I've been reading into how I might better interpolate the arguments into the URL so as to avoid injection attacks like you've mentioned. I'm reading into binding a POJO to a parameter in the docs, but I'm struggling to find how I might write tests to validate my code is interpolating the pojo as requested (short of manually going through the client logs).

How would you suggest I write tests to validate that I've written my Pojo, client and controller configured properly such that the url interpolate a pojo as an argument as I'm expecting?

from the original thread you can see that the TargetStore arg is providing a number of values, and the tcins arg is an array of id's.

I recognize this hypothetical test serves little more than to help my troubleshooting my own implementation of micronaut, but any help is very appreciated.

[yawkat]

your tests should include special characters in the input, such as & and =. then you can verify they're sent correctly.

[Jonathan-Zollinger]

thanks @yawkat !

I guess I'm lost as to how to find what URL the declarative client used when it made the request. I know the declarative client can return either a Flux<T>, Publisher<T> or a POJO, but I am farly certain none of these object include any information about the url called by the client.

in my efforts to read into this and find an answer I found this discussion from a couple years ago iterating this same thing, maybe this will explain it better than I am?

I wonder if what I'm wanting isn't possible and if I should write a low level client instead of a declarative client?

[yawkat]

i would create a fake server that verifies the urls the client is sending.