Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] attempt upgrade to axios 1.x #213

Closed
wants to merge 4 commits into from
Closed

Conversation

ricellis
Copy link
Member

An attempt to upgrade to axios 1.x.

Commit summary:

  • 8d81238 - baseline, can be dropped, just happened to be what I had locally was a different version.
  • 890e8b6 - update the package.json depenencies
    • build(deps): update axios to 1.2.0
    • This also requires new versions of:
      • typescript - to support the axios 1.x types
      • jest - to support the axios 1.x module
      • axios-cookiejar-support for axios 1.x module compatibility
    • Note: 1.2.0-alpha.1 has fixes to axios request headers that stop the retry tests from failing. Unclear on when it may move from alpha to a real release.
  • b188779 - fix axios and cookiejar support usage and tests
    • fix: axios 1.x in request-wrapper
    • Use paramsSerializer legacy compatibility serialize function with existing stringify function.
    • Correct axios-cookiejar-support import and intialization.
    • Provide backwards compatibility for jar: true option.
    • Correct mocking in request-wrapper.test.js.
    • Remove withCredentials from cookiejar.test.js.
    • Set synchronous: true on mock cookie jar.
    • Add new test for usage of real tough-cookie CookieJar.
  • 2e4618b - correct some other test errors I saw locally
    • build-request-file-object.test.js and get-content-type.test.js both make use of streams with non-existent paths to check errors. Possibly since upgrading jest these cause suites to fail with e.g.
    Error: ENOENT: no such file or directory, open '/fake/path/custom-name.env'
    Emitted 'error' event on ReadStream instance at:
        at emitErrorNT (node:internal/streams/destroy:157:8)
        at emitErrorCloseNT (node:internal/streams/destroy:122:3)
        at processTicksAndRejections (node:internal/process/task_queues:83:21) {
      errno: -2,
      code: 'ENOENT',
      syscall: 'open',
      path: '/fake/path/custom-name.env'
    
    • Since the errors are expected for these tests add a no-op handler to handle the error events without a warning
Checklist
  • npm test passes (tip: npm run lint-fix can correct most style issues)
  • tests are included
  • documentation is changed or added

@ricellis
Copy link
Member Author

AFAICT the travis failures here are caused by npm ci failing to resolve the dep tree because the 1.2.0-alpha.1 appears not to semantically satisfy cookiejar support's >=0.20.0.
I could switch to 1.1.3 and demonstrate all but the retry test passing until axios 1.2.0 solidifies if that is preferred.

@dpopp07
Copy link
Member

dpopp07 commented Nov 14, 2022

@ricellis this looks great, I appreciate all of the effort and detailed documentation. I should've been doing the same, as I've done a lot of this locally without any updates or notes on changes. However, it's a good confirmation on everything I'd worked on and you've also solved some things that were still on my to-do list. I'll go through your PR today and think about the dependency issue with the retry tests and get back to you.

I also need to do some tests for compatibility with generated code and see if any of this constitutes breaking changes for the core. This is looking great overall though, I'd love to try and merge it in the next couple of days.

@ricellis
Copy link
Member Author

However, it's a good confirmation on everything I'd worked on and you've also solved some things that were still on my to-do list

Happy to hear that it's on the same lines, do cherry-pick or copy/paste as needed into your work. We can close this PR down np.

I'll go through your PR today and think about the dependency issue with the retry tests and get back to you.

With any luck a non-alpha 1.2.0 will happen and make this problem go away. I think anything using retries will break without the type fixes in that version (not just the tests).

I also need to do some tests for compatibility with generated code and see if any of this constitutes breaking changes for the core

Yeah I wasn't sure whether we could get away with dropping the boolean on the jar option, but it seemed not a big deal to just make one to retain compatibility.

I'm not entirely sure on side-effects of the typescript version upgrade - I was definitely just focussed on getting axios working.

I do think this might not work well with Node 12, but that is out of support now anyway. If the engine does move then I'd also advocate for updating the version of the @types/node from 10 to 14 to match although I don't think it is strictly necessary usage wise atm.

@dpopp07
Copy link
Member

dpopp07 commented Nov 14, 2022

Happy to hear that it's on the same lines, do cherry-pick or copy/paste as needed into your work. We can close this PR down np.

I think I'd rather just keep your PR - it's in better shape than mine!

With any luck a non-alpha 1.2.0 will happen and make this problem go away. I think anything using retries will break without the type fixes in that version (not just the tests).

I think you're right. I ran into this problem as well. We can monitor for a release but the timeline might not be as soon as we'd like.

Yeah I wasn't sure whether we could get away with dropping the boolean on the jar option, but it seemed not a big deal to just make one to retain compatibility.

I think I'd just dropped it in my local changes but I prefer what you've done here 👍

I'm not entirely sure on side-effects of the typescript version upgrade

This is what I'm most concerned about but I had done some tests on this subject already and things seemed like they should be fine. I just want to double check.

I do think this might not work well with Node 12, but that is out of support now anyway

Yep, we have an issue on our internal board to move up the engines and update our Travis builds. We need to do that soon. Agreed that we should bump the types when we do that.

@ricellis
Copy link
Member Author

ricellis commented Nov 18, 2022

Just to add another note here on axios 1.2 and the retry issues.
The error I saw in the retry tests with axios 1.1.3 was this invalid char in header content issue. When I upgraded to 1.2.0-alpha.1 the problem resolved. Noting that this does not appear to be resolved by 1.2 for everyone so we may need to evaluate carefully.

@ricellis
Copy link
Member Author

Axios 1.2.0 is out now so I've rebased using that instead of the 1.2.0-alpha.1 and as expected this is now green in Node 14, but incompatible with Node 12.

I pushed an additional commit to move to 14 minimum and a test matrix of 14 & 16 and it looks ok now.

This also requires new versions of:
* `typescript` - to support the axios 1.x types
* `jest` - to support the axios 1.x module
* `axios-cookiejar-support` for axios 1.x module compatibility

Note: 1.2.0 has fixes to axios request headers that otherwise mean the retry tests fail.

Signed-off-by: Rich Ellis <ricellis@users.noreply.github.com>
Use `paramsSerializer` legacy compatibility `serialize` function with existing stringify function.
Correct axios-cookiejar-support import and intialization.
Provide backwards compatibility for `jar: true` option.

Correct mocking in request-wrapper.test.js.
Remove `withCredentials` from cookiejar.test.js.
Set `synchronous: true` on mock cookie jar.
Add new test for usage of real tough-cookie CookieJar.

Signed-off-by: Rich Ellis <ricellis@users.noreply.github.com>
build-request-file-object.test.js and get-content-type.test.js both make use of streams with non-existent paths to check errors. Possibly since upgrading jest these cause suites to fail with e.g.
```
Error: ENOENT: no such file or directory, open '/fake/path/custom-name.env'
Emitted 'error' event on ReadStream instance at:
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/fake/path/custom-name.env'
```

Since the errors are expected for these tests add a no-op handler to handle the error events.

Signed-off-by: Rich Ellis <ricellis@users.noreply.github.com>
Signed-off-by: Rich Ellis <ricellis@users.noreply.github.com>
@ricellis ricellis marked this pull request as ready for review November 24, 2022 12:24
@dpopp07
Copy link
Member

dpopp07 commented Nov 30, 2022

@ricellis Awesome - this looks great. Thanks again for all of your work on this. I will do a little testing to see how this plays with SDKs and ensure the changes are not "breaking" but I'd like to merge this soon

@ricellis
Copy link
Member Author

@dpopp07 is there a way to publish a snapshot build we could test too?

@dpopp07
Copy link
Member

dpopp07 commented Nov 30, 2022

@ricellis Yes, that's a great idea. I'll do that today

@dpopp07
Copy link
Member

dpopp07 commented Nov 30, 2022

@ricellis This branch can now be installed with the "beta" tag (npm i ibm-cloud-sdk-core@beta)

@dpopp07
Copy link
Member

dpopp07 commented Nov 30, 2022

@ricellis based on my testing, everything works fine but SDK developers will need to update the typescript and jest versions for their respective projects. I'm guessing the same will go for developers/users of the actual SDKs. I'm not yet sure if I want to treat that as a breaking change or not

@ricellis
Copy link
Member Author

ricellis commented Dec 1, 2022

This branch can now be installed with the "beta" tag (npm i ibm-cloud-sdk-core@beta)

Thanks, this makes testing things a lot easier.

based on my testing, everything works fine

Our SDK looks ok, but seeing some problems in other downstream projects (specifically around axios-cookiejar-support and custom https agents). I'm looking into those.

developers will need to update the typescript and jest versions for their respective projects

We run dependabot so we've been bumping our downstream dependencies regularly and they were way ahead of what was used here so I probably didn't think about this hard enough.

I suppose it may be possible to find a TS version that works that isn't as big a leap since I just moved to latest I didn't probe around to see what the minimum usable version might be. Although I suspect it will have to be a 4.x given there appeared to be a syntax change of some kind and once 3 to 4 has been crosssed there probably isn't much to be gained from older minor versions.

I don't know much about the compatibility statement between 3.x and 4.x and whether transpiling at 4.x rules out using with 3.x or whether it specifically depends on whether new features are used. I guess the break seen indicates that axios at least uses some of those 4.x features, but whether it impacts the parts of axios the core exposes could make a difference I suppose.

I'm guessing the same will go for developers/users of the actual SDKs

Yes, as above I don't know a lot abot the requirements here, but it is entirely possible that projects relying on TS 3.x will need to update to 4.x to work with types transpiled at 4.x.

@ricellis
Copy link
Member Author

ricellis commented Dec 2, 2022

seeing some problems in other downstream projects (specifically around axios-cookiejar-support and custom https agents). I'm looking into those.

AFAICT there is an incompatibility between newer versions of axios-cookiejar-support and retry-axios (and potentially some other interceptors/libs). I can't quite nail it down, but there is an open issue and I've added my thoughts there.

In the meantime I'm going to try and rollback the axios-cookiejar-support version (I bumped it based on #209, but I'm not 100% sure it was necessary). Edit: yeah it was necessary 😄

@ricellis
Copy link
Member Author

ricellis commented Dec 5, 2022

@dpopp07 I pushed another branch with a commit 220c48e that removes the dependency on axios-cookiejar-support and uses an interceptor to interact with tough-cookie instead.

Given the availability of interceptors in axios I was looking for options that avoid the dependency that is giving us trouble. Although the change is probably a bit rough around the edges, especially in terms of validating it works correctly when setting/getting multiple cookies for a single URL. I'm not sure how many other SDKs rely on cookie support, but if you want to consider making another beta from that branch/commit then I'd be happy to run our test pipelines and see if it works better for us.

@dpopp07
Copy link
Member

dpopp07 commented Dec 5, 2022

I'm not sure how many other SDKs rely on cookie support

I don't think that many do. I'll push a second beta with your changes shortly

Edit: done, beta tag updated

@ricellis
Copy link
Member Author

ricellis commented Dec 6, 2022

@dpopp07 thanks for that.

The bad news is that I see a bunch of different failures using the beta-2; the good news is that they weren't caused by my cookie change (I went back to beta-1 and they were still present). Seems like the beta-1 didn't get used in my SDK pipeline first time around (probably a bad merge on the package-lock) but was correctly used in my downstream lib pipelines which I guess explains why I saw more failures there.

Anyway the "new" failures look like some kind of unicode or encoding problem - I'll invest some time into those and see if I can move this any closer to stability, but for now it is sadly too broken to proceed.

@ricellis
Copy link
Member Author

ricellis commented Dec 6, 2022

OK so the problem is axios/axios#5296 which is fixed in axios 1.2.1 that it just so happens was released last night. That version looks ok locally so I'll try the pipelines again.

@dpopp07
Copy link
Member

dpopp07 commented Dec 6, 2022

@ricellis I updated the beta with your new changes

@ricellis
Copy link
Member Author

ricellis commented Dec 7, 2022

Thanks @dpopp07 - I couldn't see which commit made the beta-3, but it is still pulling axios 1.2.0 instead of 1.2.1.

Please could you make one from 0f65ae1? I think that has the best chance of success. If we're still having issues after that I think we'll have to think of some kind of alternative or potentially wait for axios 1.x to stabilize further.

@dpopp07
Copy link
Member

dpopp07 commented Dec 7, 2022

but it is still pulling axios 1.2.0 instead of 1.2.1

Yep, I see what happened. My mistake. I just published the new beta from that commit

@ricellis
Copy link
Member Author

ricellis commented Dec 8, 2022

Yep, I see what happened. My mistake. I just published the new beta from that commit

No worries, I appreciate the time you've been putting into releasing all these betas to help me test downstream and I didn't make it any easier with my own mistakes and having two separate branches!

beta-4 definitely uses axios 1.2.1 and in general things look much better. The bad news is that I made an error in the cookie-support code. The good news is that when I use sed to patch that error in our pipelines there are no more test failures. I've got some more soak tests running now.

@ricellis
Copy link
Member Author

ricellis commented Dec 9, 2022

Closing this, replaced by:

@ricellis ricellis closed this Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants