fix: potential bug in content type parser #4477

Uzlopak · 2022-12-21T14:47:39Z

Checklist

run npm run test and npm run benchmark
tests and/or benchmarks are included
documentation is changed or added
commit message and code follows the Developer's Certification of Origin
and the Code of conduct

jsumners

Please add a test that fails without this change.

Uzlopak · 2022-12-21T14:56:41Z

First I want to see if the CI/CD pipeline fails or not. I have locally issues with running the tests, some unrelated tests fail.

Uzlopak · 2022-12-21T15:59:02Z

Ok, it seems like my expectation was met and the codechange did not make it fail the ci pipeline.

I guess the unit test passes because it falls back to the json parser. Will provide a unit test asap.

climba03003 · 2022-12-21T17:23:55Z

I guess the unit test passes because it falls back to the json parser. Will provide a unit test asap.

Interesting, it is not failed because this.name === this.type most of the time in essence check.
And toString method do the trick.

var a = { toString() { return 'a' } }
'abc'.indexOf(a) // it is actually 0

One failing test for your changes would be addContentTypeParser('image') should match all the image content-type. But, I doubt it should be added, since developer should provide a correct content-type in first place. This behavior exist only because of the non-breaking change.

mcollina

Thanks for opening a PR! Can you please add a unit test?

Uzlopak · 2022-12-23T23:02:41Z

Added 3 Unit tests.

climba03003 · 2022-12-23T23:49:03Z

Why we would like to support a content type with leading and trailing space?
It should be a side effect of using parser.

Just a reminder, it is breaking change.

Eomm

I don't think it is breaking because the headers are trimmed and the spaces are preserved only if the user set the header value as " application/json" (with double quotes)

climba03003 · 2022-12-24T09:39:45Z

I don't think it is breaking

It is a breaking change because here change from .name check to .type check.
Please read the first comment.

Also, if we accept it. It means that #4450 will not be able to land since it will becomes a breaking change again.

ParserListItem.name is passed by user. We may trim it's value if you believe application/json (value given by developer) should be supported.

ParserListItem.type is parsed by library. They are totally two things.

climba03003 · 2022-12-24T09:52:39Z

Consider the below case currently supported, but not documented.

test('allow partial content-type', async t => {
  t.plan(1)

  const fastify = Fastify()
  fastify.removeAllContentTypeParsers()
  fastify.addContentTypeParser('json', function (request, body, done) {
    t.pass('should be called')
    done(null, body)
  })

  fastify.post('/', async () => {
    return 'ok'
  })

  await fastify.inject({
    method: 'POST',
    path: '/',
    headers: {
      'content-type': 'application/json; foo=bar; charset=utf8'
    },
    body: ''
  })
})

Eomm · 2022-12-24T10:07:48Z

I tested only the spacing cases, not the fastify.addContentTypeParser('json' 👍🏽

Uzlopak · 2022-12-24T10:21:22Z

I doubt that that works, because content-type parser will filter out media-types without / in it as invalid.

mcollina

lgtm

climba03003 · 2022-12-24T13:09:34Z

I doubt that that works, because content-type parser will filter out media-types without / in it as invalid.

Don't mix-up supported content-type and incoming content-type.
Your change affect the supported content-type (which is given by the developer and doesn't require to be a fully specified content-type.), the current state which is allowed by checking the name.
Incoming content-type never affected which require to be a valid content-type.

I still see no point on support application/json as input in addContentTypeParser. Most of the time, it should be a typo specified in code or they really mean it should be a space before or after.

climba03003

Blocking, to be sure everyone know it is a breaking change before land.
One possible plugin that break would be @fastify/multipart

Or, use this.name = contentType.trim() which does the same without breaking change.

...

dismissing for the discussion above

Uzlopak · 2023-01-03T16:13:04Z

hmm. it seems to me, that the implementation in fastify/multipart is wrong in the first place. It should be multipart/form-data and not multipart.

climba03003 · 2023-01-03T16:20:39Z

It should be multipart/form-data and not multipart.

Not really, all the multipart/* is under the same message structure. So, it is a correct assumption to use multipart only.

https://www.rfc-editor.org/rfc/rfc2046#section-5.1.1

This section defines a common syntax for subtypes of "multipart". All subtypes of "multipart" must use this syntax.

Hmm, seems like it blocked for other multipart.
https://github.com/fastify/busboy/blob/master/lib/types/multipart.js#L24

Uzlopak · 2023-01-10T23:36:00Z

Well, should we target the next branch for this PR?

mcollina · 2023-01-11T07:21:26Z

@climba03003 @Uzlopak could you reach an agreement on the couple of PRs open on this? I don't understand the conflict and what will break.

climba03003 · 2023-01-11T07:25:06Z

@climba03003 @Uzlopak could you reach an agreement on the couple of PRs open on this? I don't understand the conflict and what will break.

You can see the example in this comment.
#4477 (comment)

From what I can tell, this.name = contentType.trim() does the same with the case I mention and the test included in this PR.

mcollina · 2023-01-11T08:33:20Z

Could you send a PR with that test #4477 (comment)? So we know it's supported.

Uzlopak · 2023-01-11T09:32:58Z

I added the unit test of @climba03003 to this PR. tests pass

climba03003 · 2023-01-11T09:45:27Z

Please update the test to the following one. You are actually added a parser that would match all content-type.
When parsing a partial is failed, it return '' empty string for the .type and .indexOf('') !== -1 is always true.

test('allow partial content-type', async t => {
  t.plan(1)

  const fastify = Fastify()
  fastify.removeAllContentTypeParsers()
  fastify.addContentTypeParser('json', function (request, body, done) {
    t.pass('should be called')
    done(null, body)
  })

  fastify.post('/', async () => {
    return 'ok'
  })

  await fastify.inject({
    method: 'POST',
    path: '/',
    headers: {
      'content-type': 'application/json; foo=bar; charset=utf8'
    },
    body: ''
  })

  await fastify.inject({
    method: 'POST',
    path: '/',
    headers: {
      'content-type': 'image/jpeg'
    },
    body: ''
  })
})

fix

Uzlopak · 2023-01-11T10:18:34Z

What about now?

climba03003 · 2023-01-11T10:34:49Z

Look better, but still I believe more problem would cause than a fix when we allow trimming.

Consider the following is allowed but the developer have no idea which one should be matched (Currently, the third one).
Some plugin maybe overriding with an extra space, then the content-type parser is actually replaced without notice.

fastify.addContentTypeParser('   application/json   ', function (request, body, done) {
  done(null, body)
})
fastify.addContentTypeParser('application/json   ', function (request, body, done) {
  done(null, body)
})
fastify.addContentTypeParser('   application/json', function (request, body, done) {
  done(null, body)
})

And which one should be removed? Currently, none of them.

fastify.addContentTypeParser('   application/json   ', function (request, body, done) {
  done(null, body)
})
fastify.addContentTypeParser('application/json   ', function (request, body, done) {
  done(null, body)
})
fastify.addContentTypeParser('   application/json', function (request, body, done) {
  done(null, body)
})
fastify.removeContentTypeParser('application/json')

When you consider <space * 0-n>application/json<space * 0-n> is identical to application/json. Any methods (including, .add, .hasParser, .getParser, .remove, etc ) should treat them the same as well.

mcollina · 2023-06-07T10:19:42Z

@Uzlopak are you still pursuing this?

Uzlopak · 2023-06-07T11:15:38Z

As we never agreed on how to solve this issue, i can not finalize this PR.

Uzlopak · 2024-03-01T07:48:36Z

@gurgunday
@climba03003

Is this PR finally obsolete? Of are there still some things we can derive from this? Can we close it?

climba03003 · 2024-03-01T07:56:49Z

Is this PR finally obsolete?

Is #4509 answered your problem trying to fix in this PR.
As I remembered it was made for this PR.

Of are there still some things we can derive from this?

The next thing we need to consider is #4477 (comment)

gurgunday · 2024-03-01T08:29:15Z

The next thing we need to consider is #4477 (comment)

See #5329, this is no longer possible 🙏

I want to actually look at increasing matches though, right now it seems like a Content-Type header like text/html won't match the added parser (because we now do a startsWith check only), which is why we should be trimming it

climba03003 · 2024-03-01T09:05:03Z

See #5329, this is no longer possible 🙏

It partially addressed, you need to do the same to .hasParser and .remove.

fix potential bug

e2c9189

jsumners previously requested changes Dec 21, 2022

View reviewed changes

mcollina requested changes Dec 23, 2022

View reviewed changes

content-type isEssence-check should use ParserListItem.type

994acd8

Eomm approved these changes Dec 24, 2022

View reviewed changes

mcollina previously approved these changes Dec 24, 2022

View reviewed changes

climba03003 requested changes Dec 24, 2022

View reviewed changes

Merge branch 'main' into contentTypeParser-pot-bug

d15d2ff

Uzlopak added 2 commits January 11, 2023 10:23

Merge branch 'main' into contentTypeParser-pot-bug

a1381a4

Add unit test for content-type partial case

b59d0d0

more test cases

8e73054

fix

climba03003 mentioned this pull request Jan 11, 2023

fix: content-type mis-handling for invalid non-essence content-type #4509

Merged

4 tasks

Merge branch 'main' into contentTypeParser-pot-bug

fb886b0

Fdawgs added the bugfix Issue or PR that should land as semver patch label Mar 18, 2023

Fdawgs changed the title ~~fix potential bug in content type parser~~ fix: potential bug in content type parser Jul 2, 2023

Uzlopak mentioned this pull request Sep 4, 2023

feat: loosen content-type checking #4450

Merged

4 tasks

gurgunday mentioned this pull request Feb 25, 2024

feat: make contentTypeParser's existingParser check more strict #5329

Merged

gurgunday mentioned this pull request Mar 2, 2024

feat: align the rest of the ContentTypeParser methods with .add #5345

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: potential bug in content type parser #4477

fix: potential bug in content type parser #4477

Uzlopak commented Dec 21, 2022

jsumners left a comment

Uzlopak commented Dec 21, 2022

Uzlopak commented Dec 21, 2022

climba03003 commented Dec 21, 2022 •

edited

mcollina left a comment

Uzlopak commented Dec 23, 2022

climba03003 commented Dec 23, 2022

Eomm left a comment

climba03003 commented Dec 24, 2022 •

edited

climba03003 commented Dec 24, 2022

Eomm commented Dec 24, 2022

Uzlopak commented Dec 24, 2022

mcollina left a comment

climba03003 commented Dec 24, 2022 •

edited

climba03003 left a comment •

edited

Uzlopak commented Jan 3, 2023

climba03003 commented Jan 3, 2023 •

edited

Uzlopak commented Jan 10, 2023

mcollina commented Jan 11, 2023

climba03003 commented Jan 11, 2023

mcollina commented Jan 11, 2023

Uzlopak commented Jan 11, 2023

climba03003 commented Jan 11, 2023 •

edited

Uzlopak commented Jan 11, 2023

climba03003 commented Jan 11, 2023 •

edited

mcollina commented Jun 7, 2023

Uzlopak commented Jun 7, 2023

Uzlopak commented Mar 1, 2024

climba03003 commented Mar 1, 2024 •

edited

gurgunday commented Mar 1, 2024 •

edited

climba03003 commented Mar 1, 2024 •

edited

fix: potential bug in content type parser #4477

Are you sure you want to change the base?

fix: potential bug in content type parser #4477

Conversation

Uzlopak commented Dec 21, 2022

Checklist

jsumners left a comment

Choose a reason for hiding this comment

Uzlopak commented Dec 21, 2022

Uzlopak commented Dec 21, 2022

climba03003 commented Dec 21, 2022 • edited

mcollina left a comment

Choose a reason for hiding this comment

Uzlopak commented Dec 23, 2022

climba03003 commented Dec 23, 2022

Eomm left a comment

Choose a reason for hiding this comment

climba03003 commented Dec 24, 2022 • edited

climba03003 commented Dec 24, 2022

Eomm commented Dec 24, 2022

Uzlopak commented Dec 24, 2022

mcollina left a comment

Choose a reason for hiding this comment

climba03003 commented Dec 24, 2022 • edited

climba03003 left a comment • edited

Choose a reason for hiding this comment

Uzlopak commented Jan 3, 2023

climba03003 commented Jan 3, 2023 • edited

Uzlopak commented Jan 10, 2023

mcollina commented Jan 11, 2023

climba03003 commented Jan 11, 2023

mcollina commented Jan 11, 2023

Uzlopak commented Jan 11, 2023

climba03003 commented Jan 11, 2023 • edited

Uzlopak commented Jan 11, 2023

climba03003 commented Jan 11, 2023 • edited

mcollina commented Jun 7, 2023

Uzlopak commented Jun 7, 2023

Uzlopak commented Mar 1, 2024

climba03003 commented Mar 1, 2024 • edited

gurgunday commented Mar 1, 2024 • edited

climba03003 commented Mar 1, 2024 • edited

climba03003 commented Dec 21, 2022 •

edited

climba03003 commented Dec 24, 2022 •

edited

climba03003 commented Dec 24, 2022 •

edited

climba03003 left a comment •

edited

climba03003 commented Jan 3, 2023 •

edited

climba03003 commented Jan 11, 2023 •

edited

climba03003 commented Jan 11, 2023 •

edited

climba03003 commented Mar 1, 2024 •

edited

gurgunday commented Mar 1, 2024 •

edited

climba03003 commented Mar 1, 2024 •

edited