Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace baseUrl with prefixUrl #829

Merged
merged 9 commits into from
Sep 4, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion advanced-creation.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ const noUserAgent = got.extend({

```js
const httpbin = got.extend({
baseUrl: 'https://httpbin.org/'
prefixUrl: 'https://httpbin.org/'
szmarczak marked this conversation as resolved.
Show resolved Hide resolved
});
```

Expand Down
3 changes: 1 addition & 2 deletions migration-guides.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,6 @@ The [`timeout` option](https://github.com/sindresorhus/got#timeout) has some ext

The [`searchParams` option](https://github.com/sindresorhus/got#searchParams) is always serialized using [`URLSearchParams`](https://developer.mozilla.org/en-US/docs/Web/API/URLSearchParams) unless it's a `string`.

The [`baseUrl` option](https://github.com/sindresorhus/got#baseurl) appends the ending slash if it's not present.

There's no `maxRedirects` option. It's always set to `10`.

To use streams, just call `got.stream(url, options)` or `got(url, {stream: true, ...}`).
Expand All @@ -83,6 +81,7 @@ To use streams, just call `got.stream(url, options)` or `got(url, {stream: true,
- No `forever` option. You need to use [forever-agent](https://github.com/request/forever-agent).
- No `proxy` option. You need to [pass a custom agent](readme.md#proxies).
- No `removeRefererHeader` option. You can remove the referer header in a [`beforeRequest` hook](https://github.com/sindresorhus/got#hooksbeforeRequest):
- No `baseUrl` option. Instead, there is `prefixUrl` which appends ending slash if not present.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- No `baseUrl` option. Instead, there is `prefixUrl` which appends ending slash if not present.
- No `baseUrl` option. Instead, there is `prefixUrl` which appends ending slash if not present. If `prefixUrl` is used, it will be always prepended to the `url` argument.

Actually should we do

if (options.prefixUrl && !urlArgument.startsWith('http:') && !urlArgument.startsWith('https:') && !urlArgument.startsWith('unix:')) {
    ...
}

or let this behavior be?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My opinion has always been that input should not be able to override the host of a prefixUrl, otherwise it's not really a prefix. If such an override is needed, and it very well may be for some use cases, then I feel that full URL resolution is the answer (I.e. baseUrl). I don't think we have any business keeping a list of "special" schemes like http: where absolute URLs can override the target host. That is the job of a URL resolver, and I'd prefer to either fully embrace URL resolution or not at all.

I'm personally fine with having both prefixUrl and baseUrl. But I know Sindre felt that would be confusing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we pass a URL instance to input? I think we should ignore prefixUrl then (same for Ky).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does a URL instance change the behavior as opposed to a string? I would expect them to behave the same.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The URL instance is an absolute URL. Is there any use case for joining two absolute URLs? The string does not need to be absolute. That's the difference.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, right. I forgot that URL instances are always absolute, because that's such a silly limitation in the URL class. So, yeah, if absolute URLs are allowed to override prefixUrl, then it makes sense. However, in terms of the code, I would probably not explicitly check for URL instances, if possible. After all, what if URL starts supporting relative URLs in the future? I think they could introduce support for that without breaking too many things. And I hope they do, personally.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My opinion has always been that input should not be able to override the host of a prefixUrl

👍 (We can reconsider if we get a lot of complain about this)


```js
const gotInstance = got.extend({
Expand Down
25 changes: 9 additions & 16 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,28 +114,21 @@ Type: `object`

Any of the [`https.request`](https://nodejs.org/api/https.html#https_https_request_options_callback) options.

###### baseUrl
###### prefixUrl

Type: `string | object`

When specified, `url` will be prepended by `baseUrl`.<br>
If you specify an absolute URL, it will skip the `baseUrl`.
Type: `string | URL`

Very useful when used with `got.extend()` to create niche-specific Got instances.
When specified, `prefixUrl` will be prepended to `url`. The prefix can be any valid URL, either relative or absolute. A trailing slash `/` is optional, one will be added automatically, if needed, when joining `prefixUrl` and `url`. The `url` argument cannot start with a `/` when using this option.

Can be a string or a [WHATWG `URL`](https://nodejs.org/api/url.html#url_class_url).

Slash at the end of `baseUrl` and at the beginning of the `url` argument is optional:
Useful when used with `got.extend()1 to create niche-specific Got-instances.
szmarczak marked this conversation as resolved.
Show resolved Hide resolved

```js
await got('hello', {baseUrl: 'https://example.com/v1'});
//=> 'https://example.com/v1/hello'

await got('/hello', {baseUrl: 'https://example.com/v1/'});
//=> 'https://example.com/v1/hello'
const got = require('got');

await got('/hello', {baseUrl: 'https://example.com/v1'});
//=> 'https://example.com/v1/hello'
(async () => {
await ky('unicorn', {prefixUrl: 'https://cats.com'});
//=> 'https://cats.com/unicorn'
})();
```

###### headers
Expand Down
39 changes: 25 additions & 14 deletions source/normalize-arguments.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,12 @@ export const preNormalizeArguments = (options: Options, defaults?: Options): Nor
options.headers = lowercaseKeys(options.headers);
}

if (options.baseUrl && !options.baseUrl.toString().endsWith('/')) {
options.baseUrl += '/';
if (options.prefixUrl) {
options.prefixUrl = options.prefixUrl.toString();

if (!options.prefixUrl.toString().endsWith('/')) {
options.prefixUrl += '/';
}
}

if (is.nullOrUndefined(options.hooks)) {
Expand Down Expand Up @@ -125,7 +129,7 @@ export const normalizeArguments = (url: URLOrOptions, options: NormalizedOptions
let urlArgument: URLArgument;
if (is.plainObject(url)) {
options = {...url, ...options};
urlArgument = options.url || '';
urlArgument = options.url || {};
delete options.url;
} else {
urlArgument = url;
Expand All @@ -143,17 +147,24 @@ export const normalizeArguments = (url: URLOrOptions, options: NormalizedOptions

let urlObj: https.RequestOptions | URLOptions;
if (is.string(urlArgument)) {
if (options.baseUrl) {
if (urlArgument.startsWith('/')) {
urlArgument = urlArgument.slice(1);
}
} else {
urlArgument = urlArgument.replace(/^unix:/, 'http://$&');
if (options.prefixUrl && urlArgument.startsWith('/')) {
throw new Error('`url` must not begin with a slash when using `prefixUrl`');
}

urlObj = urlArgument || options.baseUrl ? urlToOptions(new URL(urlArgument, options.baseUrl)) : {};
if (options.prefixUrl) {
urlArgument = options.prefixUrl + urlArgument;
}

urlArgument = urlArgument.replace(/^unix:/, 'http://$&');

urlObj = urlToOptions(new URL(urlArgument));
} else if (is.urlInstance(urlArgument)) {
urlObj = urlToOptions(urlArgument);
} else if (options.prefixUrl) {
urlObj = {
...urlToOptions(new URL(options.prefixUrl)),
...urlArgument
};
} else {
urlObj = urlArgument;
}
Expand All @@ -169,12 +180,12 @@ export const normalizeArguments = (url: URLOrOptions, options: NormalizedOptions
}
}

const {baseUrl} = options;
Object.defineProperty(options, 'baseUrl', {
const {prefixUrl} = options;
Object.defineProperty(options, 'prefixUrl', {
set: () => {
throw new Error('Failed to set baseUrl. Options are normalized already.');
throw new Error('Failed to set prefixUrl. Options are normalized already.');
},
get: () => baseUrl
get: () => prefixUrl
});

let {searchParams} = options;
Expand Down
4 changes: 2 additions & 2 deletions source/utils/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ export interface Options extends Omit<https.RequestOptions, 'agent' | 'timeout'
responseType?: ResponseType;
resolveBodyOnly?: boolean;
followRedirect?: boolean;
baseUrl?: URL | string;
prefixUrl?: URL | string;
timeout?: number | Delays;
dnsCache?: Map<string, string> | Keyv | false;
url?: URL | string;
Expand All @@ -151,7 +151,7 @@ export interface NormalizedOptions extends Omit<Required<Options>, 'timeout' | '
gotTimeout: Required<Delays>;
retry: NormalizedRetryOptions;
lookup?: CacheableLookup['lookup'];
readonly baseUrl: string;
readonly prefixUrl: string;
path: string;
hostname: string;
host: string;
Expand Down
38 changes: 15 additions & 23 deletions test/arguments.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ test('throws an error if the protocol is not specified', async t => {
test('string url with searchParams is preserved', withServer, async (t, server, got) => {
server.get('/', echoUrl);

const path = '/?test=http://example.com?foo=bar';
const path = '?test=http://example.com?foo=bar';
const {body} = await got(path);
t.is(body, path);
t.is(body, `/${path}`);
});

test('options are optional', withServer, async (t, server, got) => {
Expand Down Expand Up @@ -161,7 +161,7 @@ test('accepts `url` as an option', withServer, async (t, server, got) => {
await t.notThrowsAsync(got({url: 'test'}));
});

test('can omit `url` option if using `baseUrl`', withServer, async (t, server, got) => {
test('can omit `url` option if using `prefixUrl`', withServer, async (t, server, got) => {
server.get('/', echoUrl);

await t.notThrowsAsync(got({}));
Expand Down Expand Up @@ -206,49 +206,41 @@ test('allows extra keys in `options.hooks`', withServer, async (t, server, got)
await t.notThrowsAsync(got('test', {hooks: {extra: {}}}));
});

test('`baseUrl` option works', withServer, async (t, server, got) => {
test('`prefixUrl` option works', withServer, async (t, server, got) => {
server.get('/test/foobar', echoUrl);

const instanceA = got.extend({baseUrl: `${server.url}/test`});
const {body} = await instanceA('/foobar');
t.is(body, '/test/foobar');
});

test('accepts WHATWG URL as the `baseUrl` option', withServer, async (t, server, got) => {
server.get('/test/foobar', echoUrl);

const instanceA = got.extend({baseUrl: new URL(`${server.url}/test`)});
const {body} = await instanceA('/foobar');
const instanceA = got.extend({prefixUrl: `${server.url}/test`});
const {body} = await instanceA('foobar');
t.is(body, '/test/foobar');
});

test('backslash in the end of `baseUrl` option is optional', withServer, async (t, server) => {
test('accepts WHATWG URL as the `prefixUrl` option', withServer, async (t, server, got) => {
server.get('/test/foobar', echoUrl);

const instanceA = got.extend({baseUrl: `${server.url}/test/`});
const {body} = await instanceA('/foobar');
const instanceA = got.extend({prefixUrl: new URL(`${server.url}/test`)});
const {body} = await instanceA('foobar');
t.is(body, '/test/foobar');
});

test('backslash in the beginning of `url` is optional when using `baseUrl` option', withServer, async (t, server) => {
test('backslash in the end of `prefixUrl` option is optional', withServer, async (t, server) => {
server.get('/test/foobar', echoUrl);

const instanceA = got.extend({baseUrl: `${server.url}/test`});
const instanceA = got.extend({prefixUrl: `${server.url}/test/`});
const {body} = await instanceA('foobar');
t.is(body, '/test/foobar');
});

test('throws when trying to modify `baseUrl` after options got normalized', async t => {
test('throws when trying to modify `prefixUrl` after options got normalized', async t => {
const instanceA = got.create({
methods: [],
options: {baseUrl: 'https://example.com'},
options: {prefixUrl: 'https://example.com'},
handler: (options, next) => {
options.baseUrl = 'https://google.com';
options.prefixUrl = 'https://google.com';
return next(options);
}
});

await t.throwsAsync(instanceA('/'), 'Failed to set baseUrl. Options are normalized already.');
await t.throwsAsync(instanceA(''), 'Failed to set prefixUrl. Options are normalized already.');
});

test('throws if the `searchParams` value is invalid', async t => {
Expand Down
2 changes: 1 addition & 1 deletion test/cache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ test('doesn\'t cache response when received HTTP error', withServer, async (t, s

test('DNS cache works', withServer, async (t, _server, got) => {
const map = new Map();
await t.notThrowsAsync(got('https://example.com', {dnsCache: map}));
await t.notThrowsAsync(got('https://example.com', {dnsCache: map, prefixUrl: null}));
sindresorhus marked this conversation as resolved.
Show resolved Hide resolved

t.is(map.size, 1);
});
Expand Down
21 changes: 12 additions & 9 deletions test/create.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,12 @@ test('extend keeps the old value if the new one is undefined', t => {
});

test('extend merges URL instances', t => {
const a = got.extend({baseUrl: new URL('https://example.com')});
const b = a.extend({baseUrl: '/foo'});
t.is(b.defaults.options.baseUrl.toString(), 'https://example.com/foo/');
// @ts-ignore Custom instance.
const a = got.extend({custom: new URL('https://example.com')});
// @ts-ignore Custom instance.
const b = a.extend({custom: '/foo'});
// @ts-ignore Custom instance.
t.is(b.defaults.options.custom.toString(), 'https://example.com/foo');
});

test('create', withServer, async (t, server) => {
Expand Down Expand Up @@ -128,8 +131,8 @@ test('hooks are merged on got.extend()', t => {
test('custom endpoint with custom headers (extend)', withServer, async (t, server) => {
server.all('/', echoHeaders);

const instance = got.extend({headers: {unicorn: 'rainbow'}, baseUrl: server.url});
const headers = await instance('/').json<TestReturn>();
const instance = got.extend({headers: {unicorn: 'rainbow'}, prefixUrl: server.url});
const headers = await instance('').json<TestReturn>();
t.is(headers.unicorn, 'rainbow');
t.not(headers['user-agent'], undefined);
});
Expand All @@ -138,7 +141,7 @@ test('no tampering with defaults', t => {
const instance = got.create({
handler: got.defaults.handler,
options: got.mergeOptions(got.defaults.options, {
baseUrl: 'example/'
prefixUrl: 'example/'
})
});

Expand All @@ -149,11 +152,11 @@ test('no tampering with defaults', t => {

// Tamper Time
t.throws(() => {
instance.defaults.options.baseUrl = 'http://google.com';
instance.defaults.options.prefixUrl = 'http://google.com';
});

t.is(instance.defaults.options.baseUrl, 'example/');
t.is(instance2.defaults.options.baseUrl, 'example/');
t.is(instance.defaults.options.prefixUrl, 'example/');
t.is(instance2.defaults.options.prefixUrl, 'example/');
});

test('defaults can be mutable', t => {
Expand Down
4 changes: 2 additions & 2 deletions test/helpers/with-server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ export default async (t, run) => {
});

// @ts-ignore Ignore errors for extending got, for the tests
const preparedGot = got.extend({baseUrl: server.url, avaTest: t.title});
const preparedGot = got.extend({prefixUrl: server.url, avaTest: t.title});
// @ts-ignore Ignore errors for extending got, for the tests
preparedGot.secure = got.extend({baseUrl: server.sslUrl, avaTest: t.title});
preparedGot.secure = got.extend({prefixUrl: server.sslUrl, avaTest: t.title});

server.hostname = (new URL(server.url)).hostname;
server.sslHostname = (new URL(server.sslUrl)).hostname;
Expand Down
2 changes: 1 addition & 1 deletion test/hooks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,7 @@ test('beforeError allows modifications', async t => {
test('does not break on `afterResponse` hook with JSON mode', withServer, async (t, server, got) => {
server.get('/foobar', echoHeaders);

await t.notThrowsAsync(got('/', {
await t.notThrowsAsync(got('', {
hooks: {
afterResponse: [
(response, retryWithMergedOptions) => {
Expand Down
4 changes: 2 additions & 2 deletions test/merge-instances.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ test('merging instances', withServer, async (t, server) => {
server.get('/', echoHeaders);

const instanceA = got.extend({headers: {unicorn: 'rainbow'}});
const instanceB = got.extend({baseUrl: server.url});
const instanceB = got.extend({prefixUrl: server.url});
const merged = got.mergeInstances(instanceA, instanceB);

const headers = await merged('/').json<TestReturn>();
const headers = await merged('').json<TestReturn>();
t.is(headers.unicorn, 'rainbow');
t.not(headers['user-agent'], undefined);
});
Expand Down
2 changes: 1 addition & 1 deletion test/progress.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ test('download progress - missing total size', withServer, async (t, server, got

const events = [];

await got('/').on('downloadProgress', event => events.push(event));
await got('').on('downloadProgress', event => events.push(event));

checkEvents(t, events);
});
Expand Down
2 changes: 1 addition & 1 deletion test/stream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ test('has error event', withServer, async (t, server, got) => {
});

test('has error event #2', withServer, async (t, _server, got) => {
const stream = got.stream('http://doesntexist');
const stream = got.stream('http://doesntexist', {prefixUrl: null});
await t.throwsAsync(pEvent(stream, 'response'), {code: 'ENOTFOUND'});
});

Expand Down