feat: support concurrent test runs via "server.boundary" #2000

kettanaito · 2024-01-26T12:51:05Z

Fixes Support concurrent runs in Node.js #474
Closes test(concurrency): add a failing concurrency test #1370
A bit about how this works: Support concurrent runs in Node.js #474 (comment)
Documented in document "server.boundary()" API mswjs.io#356

Changes

The server.boundary() API is a Node.js-only API that provides encapsulation to whichever given callback function by executing it within an AsyncLocalStorage context. This API is designed to support using MSW in concurrent test runs since no changes to request handlers (e.g. using server.use()) will ever leave the boundary function.

server.boundary() has its practical use outside of testing as well. You can use it in any concurrent logic that performs network request, if you wish to handle those requests differently on a per-boundary basis.

Usage

You must wrap each test function in a concurrent test suite in the server.boundary() function to ensure proper isolation.

it.concurrent('one', server.boundary(async () => {
  await fetch('/resource') // 200 OK
}))

it.concurrent('two', server.boundary(async () => {
  // This request handler override will only affect the network
  // calls that happen within this server boundary. Neat!
  server.use(http.get('/resource', () => HttpResponse.error()))
  await fetch('/resource') // Failed to fetch
}))

Server boundary can also be used anywhere else in your application to affect the network only within the given function scope. For example, you can debug what network requests a subtree of your application makes:

server.boundary(() => {
  http.all('*', ({ request }) => console.log(request.method, request.url))
  subtree()
})()

Todos

Test server.boundary(). Every behavior must be verified in tests.
Ensure React Native build isn't broken. I doubt there's node:async_events there, so a dependency on it in SetupServerApi may throw on runtime. 4a1c983
Document server.boundary()

mattcosta7 · 2024-01-26T16:04:40Z

src/node/SetupServerApi.ts

+   * `server.use()`) will be scoped to this boundary only.
+   * @param callback A function to run (e.g. a test)
+   */
+  public boundary<Fn extends (...args: Array<unknown>) => unknown>(


neat!

should every call to use create some kind of boundary internally - can we avoid leaking that to the end user? (sorry if this is still early)

Yes and no. We can use .enterWith() from AsyncLocalStorage on each server.use(), and while it would solve some use cases, it would break the other.

The problem is that relying on async storage internally makes it hard to support sequential tests. Here's an example:

const server = setupServer() beforeAll(() => { // This will have a different execution id... server.use(...someHandlers) server.listen() }) it('test', () => { // ...than this. server.use(...anotherHandlers) // So the request handling logic that needs the list // of all handlers won't be able to access the context // (and the handlers) introduced in "beforeAll". fetch() })

The issue here is that the context created by the server.use() call in beforeAll will be different and inaccessible by the context created by sever.use() in the test ( they have different execution async ids).

I also have concerns that using .enterWith() on each server.use() call creates nested execution contexts (imagine you have multiple .use() calls in a single test), which may have negative impact on runtime performance.

It is for this reason I'm proposing an explicit server.boundary() API. This API is effectively a wrapper around .run() function from async storage and it doesn't create a new execution context (at least as far as I understand). In any case, since it's not bound to individual .use() calls, you can have as many calls as you wish within a single boundary and they will remain within the same execution context.

that makes sense. it's too bad that this leaks though, but I don't think that's a big deal.

I actually think this is very good that it leaks. It makes a server boundary a conscious choice. Just as running tests in parallel is a conscious choice (no testing framework does it by default).

Server boundary is also incredibly powerful outside of testing too! Imagine you want to quickly fail all requests coming from a sub-tree in your app:

server.boundary(() => { http.all('*', () => HttpResponse.error()) subtree() })()

This will fail all network requests within subtree() but have no effect on the network outside of this boundary. Looks like a neat debugging tool to have! (And since handlers are "transparent" by default, it makes it twice as useful)

kettanaito · 2024-01-26T16:13:45Z

package.json

@@ -52,7 +52,7 @@
    "check:exports": "node \"./config/scripts/validate-esm.js\"",
    "test": "pnpm test:unit && pnpm test:node && pnpm test:browser && pnpm test:native",
    "test:unit": "vitest",
-    "test:node": "vitest --config=./test/node/vitest.config.ts",
+    "test:node": "vitest run --config=./test/node/vitest.config.ts",


Note: our integration tests in Node.js cannot be run in watch mode because we have one test that builds the library. It triggers the entire test run anew.

kettanaito · 2024-01-26T16:14:06Z

src/browser/setupWorker/setupWorker.ts

@@ -58,8 +58,10 @@ export class SetupWorkerApi
      isMockingEnabled: false,
      startOptions: null as any,
      worker: null,
+      currentHandlers: () => {


I'm making this a getter the way it was always supposed to be.

src/core/SetupApi.ts

kettanaito · 2024-01-26T16:46:25Z

If this API makes it, I will write a blog post about it. The more I throw different use cases at it, the more I love it!

christoph-fricke · 2024-02-01T09:28:50Z

This looks really interesting! Recently, I have been thinking about use-cases for AsyncLocalStorage, mostly using it as a dependency context mechanism. server.boundary looks like a really good use case for it. I like it!

Personally, I mostly use MSW to mock network requests in Playwright tests. My setups use either test or worker fixtures to init and reset MSW, depending on requirements. Workers are Playwright's concept for concurrent tests, with each worker being a different process. Since tests and their handlers are already isolated through workers, I have been wondering whether or not server.boundary is relevant when using MSW in Playwright. Don't get me wrong - I like it and think it is useful for MSW in general. But is it needed when using MSW in Playwright? :)

kettanaito · 2024-02-01T10:17:11Z

@christoph-fricke, thank you so much for sharing your feedback!

Since tests and their handlers are already isolated through workers, I have been wondering whether or not server.boundary is relevant when using MSW in Playwright

As far as I remember, Playwright uses workers to parallelize test files but workers have no effect on the actual browser runtime. When you use MSW with Playwright, you enable it as a part of the browser's runtime, either in your application or in the browser directly via Playwright fixtures.

With Playwright, you achieve network isolation because every tab is a separate runtime. This means that worker.use() in Tab 1 has no effect on the network in Tab 2 because those two runtimes of the same app has registered the same worker, and keep their own currentHandlers list in-memory (which is not shared).

This is why server.boundary() is a Node.js-specific API. The browser runtime has no such problem and there's also no AsyncLocalStorage in the browser.

mattcosta7 · 2024-02-08T20:05:20Z

@christoph-fricke, thank you so much for sharing your feedback!

Since tests and their handlers are already isolated through workers, I have been wondering whether or not server.boundary is relevant when using MSW in Playwright

As far as I remember, Playwright uses workers to parallelize test files but workers have no effect on the actual browser runtime. When you use MSW with Playwright, you enable it as a part of the browser's runtime, either in your application or in the browser directly via Playwright fixtures.

With Playwright, you achieve network isolation because every tab is a separate runtime. This means that worker.use() in Tab 1 has no effect on the network in Tab 2 because those two runtimes of the same app has registered the same worker, and keep their own currentHandlers list in-memory (which is not shared).

This is why server.boundary() is a Node.js-specific API. The browser runtime has no such problem and there's also no AsyncLocalStorage in the browser.

I think it depends?

You can use playwright, and mock the network from the browser with service workers (which I agree shouldn't need this).

But you can also intercept network requests from the playwright node process, in which case this could be used in playwright tests too - so it depends how/where you're applying mocks in your playwright configuration

mattcosta7

I'm really liking this. What testing/next steps do we need to push it forward?

kettanaito · 2024-02-08T20:10:37Z

But you can also intercept network requests from the playwright node process

I'm missing context on this. What is "playwright node process"? I believe Playwright has two parts:

Node.js process, which is your test. It establishes a message channel with the browser runtime.
Browser process, the actual running browser runtime.

If I understand you right, you mean you can use setupServer in the Playwright test to intercept requests you make in that test... But I fail to see when you would do those requests. If you use Playwright, your tested code runs in the browser process. Otherwise, you don't really need Playwright.

kettanaito · 2024-02-08T20:12:06Z

I'm really liking this. What testing/next steps do we need to push it forward?

It's fully ready. I've written a blog post on this as well to introduce this change and dive a bit into its technical implementation. The next step is to wait for my wife to finish the illustration for the blog post and we can get it released! This is likely to happen next Monday. Exciting!

Until then, I can use some feedback on the Server Boundary API docs. If you have a minute, Matt, could you share your opinion on that page? Is everything explained clearly?

mattcosta7 · 2024-02-08T20:12:47Z

But you can also intercept network requests from the playwright node process

I'm missing context on this. What is "playwright node process"? I believe Playwright has two parts:

Node.js process, which is your test. It establishes a message channel with the browser runtime.

Browser process, the actual running browser runtime.

If I understand you right, you mean you can use setupServer in the Playwright test to intercept requests you make in that test... But I fail to see when you would do those requests. If you use Playwright, your tested code runs in the browser process. Otherwise, you don't really need Playwright.

Yeah - you can use playwright's request interception to intercept the request from the node side instead of from the browser side. I'm not really sure why you would want to do that instead of just letting the service worker handle it - but you could do that https://playwright.dev/docs/mock

kettanaito · 2024-02-08T20:15:00Z

@mattcosta7, but that's just API semantics. Regardless of what you choose to use in Playwright, the only environment where those requests actually happen is still the browser. Even with the .route() API in Playwright, no requests will happen in the Node.js process. It's just a binding that reports requests Playwright intercepts in the browser and forwards them to that binding. You wouldn't need the server boundary in this scenario.

mattcosta7 · 2024-02-08T20:16:31Z

@mattcosta7, but that's just API semantics. Regardless of what you choose to use in Playwright, the only environment where those requests actually happen is still the browser. Even with the .route() API in Playwright, no requests will happen in the Node.js process. It's just a binding that reports requests Playwright intercepts in the browser and forwards them to that binding. You wouldn't need the server boundary in this scenario.

Ahh - interesting, this could have been a misunderstanding on my part!

mattcosta7 · 2024-02-08T20:17:34Z

I'm really liking this. What testing/next steps do we need to push it forward?

It's fully ready. I've written a blog post on this as well to introduce this change and dive a bit into its technical implementation. The next step is to wait for my wife to finish the illustration for the blog post and we can get it released! This is likely to happen next Monday. Exciting!

Until then, I can use some feedback on the Server Boundary API docs. If you have a minute, Matt, could you share your opinion on that page? Is everything explained clearly?

Will do - mostly likely tomorrow morning

kettanaito · 2024-02-08T20:19:26Z

Ahh - interesting, this could have been a misunderstanding on my part!

No worries! The node/browser bindings can get quite confusing. Afaik, everything you access from Playwright is created just for you. All of those are bindings that communicate with the browser process.

mattcosta7 · 2024-02-09T16:33:45Z

@kettanaito I think the docs looks good!

this is a minor nit/question:

I wonder if we should / need to call out that while the handlers are scoped to the boundary, any user objects defined outside of the boundary are not. This might be not-necessary, since it's basically just how scope works in the language, but I wonder if we'll receive issues like

const someThing = new SomeThing()
const server = setupServer()

test('', () => {
   server.boundary(() => {
       server.use(
           http.post('https://example.com/login', () => {
               sometThing.field = 1
               return new HttpResponse.json(someThing, { status: 500 })
            })
       )
   })
   expect(someThing.field).toBe(1)
})

test('', () => {
   server.boundary(() => {
       server.use(
           http.get('https://example.com/login', () => {
               return new HttpResponse.json(someThing, { status: 500 })
            })
       )
   })
   expect(someThing.field).toBeUndefined()
})

super trivial example, but run concurrently these can race easily it seems likely state could leak into boundaries this way, and it might be good to call out that users should avoid leaking state when using boundaries and concurrency

I think this is generally just an indicator of how they should be setting up their tests to handle concurrency, but it might be worth warning about too.

kettanaito · 2024-02-09T17:57:03Z

@mattcosta7, that's a good point. Let's omit this on the ground of being common knowledge but if we ever get an issue about someone expecting that from the server boundary, we can add a notice to the docs.

The post is ready, planned to be released on Monday. 🚀

christoph-fricke · 2024-02-09T21:02:23Z

@mattcosta7, but that's just API semantics. Regardless of what you choose to use in Playwright, the only environment where those requests actually happen is still the browser. Even with the .route() API in Playwright, no requests will happen in the Node.js process. It's just a binding that reports requests Playwright intercepts in the browser and forwards them to that binding. You wouldn't need the server boundary in this scenario.

@kettanaito That only right for CSR apps. I prefer using Playwright's network mocking API and delegate requests to MSW as described by you both. Kinda what playwright-msw is doing, as I prefer to test against a production build with zero test related code in it.

However, testing SSR apps requires mocking requests that happen in a Node.js process. A colleague and I found quite a nice solution for testing Remix apps, which fully isolates tests with a mocked network thanks to Playwright's and MSW's flexible APIs. The trick is to use MSW Node.js API to intercept requests in worker processes and create the remix server in the same worker. Playwright fixtures are ideal for achieving this. Thanks to the strong worker isolation of Playwright (OS processes), this pattern still does not need server.boundary(). A rough POC implementation of this approach can be found here: https://github.com/luchsamapparat/remix-playwright-msw/blob/master/tests/fixture.ts

kettanaito · 2024-02-11T12:35:25Z

@christoph-fricke, makes sense. We are also exploring setupRemoteServer for native server-side network interception in MSW (see #1617).

kettanaito · 2024-02-12T11:06:27Z

Released: v2.2.0 🎉

This has been released in v2.2.0!

📄 Release notes
📦 npm package

Make sure to always update to the latest version (npm i msw@latest) to get the newest features and bug fixes.

Predictable release automation by @ossjs/release.

HristoKolev · 2024-04-01T18:10:53Z

Just FYI this fixes a bug where server.resetHandlers would act as server.use and not reset handlers. I recently updated msw (node) from 2.0.11 to 2.2.13 and that caused a bunch of tests to fail because resetHandlers started to work correctly. I tracked it down to 2.2.0 and probably this PR.

kettanaito added 2 commits January 26, 2024 13:44

feat(wip): concurrent runs

9210781

feat: add "server.boundary" for concurrent tests

c4447bf

kettanaito changed the title ~~feat: support concurrent mode in nodejs~~ feat: support concurrent mode via "server.boundary" Jan 26, 2024

mattcosta7 reviewed Jan 26, 2024

View reviewed changes

kettanaito commented Jan 26, 2024

View reviewed changes

src/core/SetupApi.ts Show resolved Hide resolved

kettanaito requested a review from mattcosta7 January 26, 2024 16:18

kettanaito mentioned this pull request Jan 26, 2024

Support concurrent runs in Node.js #474

Closed

fix(setupWorker): use "getRequestHandlers" method consistently

7f30fae

kettanaito added the needs:discussion label Jan 26, 2024

kettanaito mentioned this pull request Feb 3, 2024

document "server.boundary()" API mswjs/mswjs.io#356

Merged

kettanaito added the release candidate label Feb 3, 2024

kettanaito marked this pull request as ready for review February 3, 2024 11:53

kettanaito added needs:tests and removed release candidate needs:discussion labels Feb 3, 2024

kettanaito self-assigned this Feb 3, 2024

kettanaito added 4 commits February 3, 2024 16:32

Merge branch 'main' into feat/concurrent-runs

a145591

test: add handlers propagation tests

6c8c208

test: add callback args and return tests

ef28f7e

docs: move boundary jsdoc to glossary

5d77008

kettanaito added release candidate and removed needs:tests labels Feb 5, 2024

chore: separate SetupServerApi for node and native

5106393

kettanaito force-pushed the feat/concurrent-runs branch from 4a1c983 to 5106393 Compare February 5, 2024 18:51

Merge branch 'main' into feat/concurrent-runs

297cd18

mattcosta7 reviewed Feb 8, 2024

View reviewed changes

kettanaito changed the title ~~feat: support concurrent mode via "server.boundary"~~ feat: support concurrent test runs via "server.boundary" Feb 8, 2024

mattcosta7 approved these changes Feb 9, 2024

View reviewed changes

kettanaito merged commit 450e7bc into main Feb 12, 2024
11 checks passed

kettanaito deleted the feat/concurrent-runs branch February 12, 2024 09:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support concurrent test runs via "server.boundary" #2000

feat: support concurrent test runs via "server.boundary" #2000

kettanaito commented Jan 26, 2024 •

edited

mattcosta7 Jan 26, 2024

kettanaito Jan 26, 2024 •

edited

mattcosta7 Jan 26, 2024

kettanaito Jan 26, 2024 •

edited

kettanaito Jan 26, 2024

kettanaito Jan 26, 2024

kettanaito commented Jan 26, 2024

christoph-fricke commented Feb 1, 2024 •

edited

kettanaito commented Feb 1, 2024

mattcosta7 commented Feb 8, 2024

mattcosta7 left a comment

kettanaito commented Feb 8, 2024

kettanaito commented Feb 8, 2024

mattcosta7 commented Feb 8, 2024

kettanaito commented Feb 8, 2024

mattcosta7 commented Feb 8, 2024

mattcosta7 commented Feb 8, 2024

kettanaito commented Feb 8, 2024

mattcosta7 commented Feb 9, 2024 •

edited

kettanaito commented Feb 9, 2024

christoph-fricke commented Feb 9, 2024

kettanaito commented Feb 11, 2024

kettanaito commented Feb 12, 2024

HristoKolev commented Apr 1, 2024

feat: support concurrent test runs via "server.boundary" #2000

feat: support concurrent test runs via "server.boundary" #2000

Conversation

kettanaito commented Jan 26, 2024 • edited

Changes

Usage

Todos

mattcosta7 Jan 26, 2024

Choose a reason for hiding this comment

kettanaito Jan 26, 2024 • edited

Choose a reason for hiding this comment

mattcosta7 Jan 26, 2024

Choose a reason for hiding this comment

kettanaito Jan 26, 2024 • edited

Choose a reason for hiding this comment

kettanaito Jan 26, 2024

Choose a reason for hiding this comment

kettanaito Jan 26, 2024

Choose a reason for hiding this comment

kettanaito commented Jan 26, 2024

christoph-fricke commented Feb 1, 2024 • edited

kettanaito commented Feb 1, 2024

mattcosta7 commented Feb 8, 2024

mattcosta7 left a comment

Choose a reason for hiding this comment

kettanaito commented Feb 8, 2024

kettanaito commented Feb 8, 2024

mattcosta7 commented Feb 8, 2024

kettanaito commented Feb 8, 2024

mattcosta7 commented Feb 8, 2024

mattcosta7 commented Feb 8, 2024

kettanaito commented Feb 8, 2024

mattcosta7 commented Feb 9, 2024 • edited

kettanaito commented Feb 9, 2024

christoph-fricke commented Feb 9, 2024

kettanaito commented Feb 11, 2024

kettanaito commented Feb 12, 2024

Released: v2.2.0 🎉

HristoKolev commented Apr 1, 2024

kettanaito commented Jan 26, 2024 •

edited

kettanaito Jan 26, 2024 •

edited

kettanaito Jan 26, 2024 •

edited

christoph-fricke commented Feb 1, 2024 •

edited

mattcosta7 commented Feb 9, 2024 •

edited