feat: worker pool that can execute tasks on all workers #32120

pieh · 2021-06-25T12:11:05Z

Description

This swaps jest-worker with our own worker pool that will be adding features we need for PQR. Initially it has feature parity with jest-worker (for the features we do use) + ability to execute certain "tasks" on all workers (needed for bootstrapping workers for PQR - jest-worker setup was too limiting for us - we need greater control).

In future PRs there will be added support for messaging between main and workers.

Perf

Not hugely impactful for us, but I wanted to make sure worker pool performance wouldn't regress I used https://github.com/facebook/jest/tree/master/packages/jest-worker/src/__performance_tests__ and adjusted it a bit (https://gist.github.com/pieh/9c3f6dbdd3acafa5a8811418eca4d356) and compared gatsby-worker against jest-worker@27.0.2 (latest as of writing of this comment - note that we currently use older version).

Task that does nothing 100000 times

➜  __performance_tests__ git:(pqr/allow-execute-fn-on-all-workers) ✗ node --expose-gc test.js empty 100000                          
---------------------------------------------------------------------------
jest-worker: { globalTime: 1998.6634929180145, processingTime: 1963.4644238948822 }
gatsby-worker: { globalTime: 1699.7716178894043, processingTime: 1680.174674987793 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2131.742655992508, processingTime: 2099.079083919525 }
gatsby-worker: { globalTime: 1667.6132881641388, processingTime: 1639.6259500980377 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2059.533210992813, processingTime: 2031.5315339565277 }
gatsby-worker: { globalTime: 1617.8644468784332, processingTime: 1593.7371819019318 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 1988.8901131153107, processingTime: 1961.373533964157 }
gatsby-worker: { globalTime: 1653.4446740150452, processingTime: 1631.8277361392975 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2129.582818031311, processingTime: 2103.122942209244 }
gatsby-worker: { globalTime: 1700.000137090683, processingTime: 1678.7925159931183 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2058.744973897934, processingTime: 2031.553738117218 }
gatsby-worker: { globalTime: 1638.2301177978516, processingTime: 1613.9939517974854 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 1980.671779870987, processingTime: 1953.9795258045197 }
gatsby-worker: { globalTime: 1631.1066999435425, processingTime: 1608.66081905365 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 1998.1074059009552, processingTime: 1973.3088948726654 }
gatsby-worker: { globalTime: 1648.0316200256348, processingTime: 1622.8740689754486 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2051.6042828559875, processingTime: 2021.7381830215454 }
gatsby-worker: { globalTime: 1632.7712378501892, processingTime: 1612.0683648586273 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 2149.560436964035, processingTime: 2121.764895915985 }
gatsby-worker: { globalTime: 1654.3145129680634, processingTime: 1633.308058977127 }
---------------------------------------------------------------------------
total gatsby-worker: { gWGT: 16543.148352622986, gWPT: 16315.063322782516 }
total jest-worker: { jWGT: 20547.101170539856, jWPT: 20260.91675567627 }
---------------------------------------------------------------------------
global time: gatsby-worker over 100000 calls was faster by % 19.486704156874847
processing time: gatsby-worker over 100000 calls was faster by % 19.47519690484039

Task that calculate PI 10000 times in workers

➜  __performance_tests__ git:(pqr/allow-execute-fn-on-all-workers) ✗ node --expose-gc test.js loadTest 10000                        
---------------------------------------------------------------------------
jest-worker: { globalTime: 961.0926420688629, processingTime: 922.5083830356598 }
gatsby-worker: { globalTime: 777.6850469112396, processingTime: 758.6562309265137 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 854.1666779518127, processingTime: 825.3946840763092 }
gatsby-worker: { globalTime: 695.6478898525238, processingTime: 676.7463660240173 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 729.9372169971466, processingTime: 707.9941308498383 }
gatsby-worker: { globalTime: 662.4068939685822, processingTime: 642.3461120128632 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 726.789785861969, processingTime: 701.6343488693237 }
gatsby-worker: { globalTime: 890.4017338752747, processingTime: 865.6464939117432 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 723.6333639621735, processingTime: 699.9333298206329 }
gatsby-worker: { globalTime: 651.1934118270874, processingTime: 630.4632089138031 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 818.4343338012695, processingTime: 795.8016359806061 }
gatsby-worker: { globalTime: 669.1646339893341, processingTime: 645.9727358818054 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 712.160159111023, processingTime: 686.5256531238556 }
gatsby-worker: { globalTime: 654.4234750270844, processingTime: 635.5800518989563 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 757.0051820278168, processingTime: 731.649176120758 }
gatsby-worker: { globalTime: 676.8136041164398, processingTime: 653.058874130249 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 731.8604440689087, processingTime: 706.2469630241394 }
gatsby-worker: { globalTime: 742.6872208118439, processingTime: 724.3149909973145 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 716.2216629981995, processingTime: 693.820986032486 }
gatsby-worker: { globalTime: 731.365779876709, processingTime: 708.1000800132751 }
---------------------------------------------------------------------------
total gatsby-worker: { gWGT: 7151.789690256119, gWPT: 6940.885144710541 }
total jest-worker: { jWGT: 7731.301468849182, jWPT: 7471.509290933609 }
---------------------------------------------------------------------------
global time: gatsby-worker over 10000 calls was faster by % 7.495656209087455
processing time: gatsby-worker over 10000 calls was faster by % 7.10196729417121

With example benchmarks gatsby-worker seems a bit faster (~7% when calculating PI 10000 times and ~19% when doing no-op task 100000 times), but realistically we won't see impact of it in real life as our task are really heavy and we usually will have much less tasks than 10000 (so any worker pool overhead is dwarfed by actual tasks).

Documentation

In new gatsby-worker utility package README.md

Related Issues

[ch33219]

LekoArts · 2021-06-28T13:41:12Z

packages/gatsby-worker/src/__tests__/integration.ts

+      expect(returnValue).toBeArrayOfSize(numWorkers)
+      // turns sync function into async
+      expect(returnValue).toSatisfyAll(isPromise)


packages/gatsby-worker/src/__tests__/integration.ts

packages/gatsby-worker/src/child.ts

LekoArts · 2021-06-29T07:18:38Z

packages/gatsby-worker/package.json

+{
+  "name": "gatsby-worker",
+  "description": "Utility to create worker pools",
+  "version": "0.0.0-next.0",


I think we'll want to start this at 1.0.0?

Early on I would prefer to keep it <1 as we might quickly introduce breaking changes as we start using worker pool and discover new needs or APIs (that's what <1 is for). 1 is when API becomes actually stable enough

Curious how this will play with lerna minor bump - I think it should do 0.1.0-next.0 for the next branch cut? If so, this sounds good to me.

https://github.com/gatsbyjs/gatsby/commits/master/packages/gatsby-plugin-graphql-config/package.json is example of 0.X.Y package we have and it does bump minor for each release. We want gatsby-worker to be actual dep only of gatsby core package - if we will need it in gatsby-cli for structured logs, we will import it from gatsby/internal there. Messaging (but this will only need messaging and not queueing up tasks)

packages/gatsby-worker/src/index.ts

packages/gatsby-worker/src/task-queue.ts

packages/gatsby-worker/tsconfig.json

packages/gatsby/src/utils/worker/__tests__/test-helpers/create-test-worker.ts

… (preperation for messaging API)

vladar

I think we must also replace this usage of JEST_WORKER_ID:

gatsby/packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts

Line 12 in 7c343ff

process.env.FORCE_TEST_DATABASE_ID ?? process.env.JEST_WORKER_ID

vladar · 2021-06-29T09:37:29Z

packages/gatsby-worker/src/task-queue.ts

+      // if node is not the head then it will have .prev
+      taskNode.prev!.next = taskNode.next


Should we also re-assign taskNode.next.prev? Is it possible that it will retain the task through this link?

Yes, you are correct - damn basic data structures ;)

done in ca74834

pieh · 2021-06-29T10:28:59Z

I think we must also replace this usage of JEST_WORKER_ID:

gatsby/packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts

Line 12 in 7c343ff

process.env.FORCE_TEST_DATABASE_ID ?? process.env.JEST_WORKER_ID

Similar as #32120 (comment) - we actually use JEST_WORKER_ID only in test scenarios - and there we need it so parallel running tests don't try to use same db, so JEST_WORKER_ID controlling it is correct (even tho quite confusing indeed - I will add some inline comment about usage of it)

…en tho we ourselves will be using gatsby-worker

LekoArts

👍

vladar

Really great stuff 👍

pieh added the topic: core Relates to Gatsby's core (e.g. page loading, reporter, state machine) label Jun 25, 2021

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jun 25, 2021

pieh removed the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jun 25, 2021

pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 845a318 to 4d4272a Compare June 25, 2021 15:13

pieh marked this pull request as ready for review June 25, 2021 15:31

pieh mentioned this pull request Jun 25, 2021

[do-not-merge] exploring alternative to jest-worker for PQR #31941

Closed

pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 4d4272a to bf4c08b Compare June 28, 2021 09:24

feat: create gatsby-worker package

bc67f4f

pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from bf4c08b to ad42d34 Compare June 28, 2021 09:28

pieh added 3 commits June 28, 2021 11:57

chore(gatsby): convert ssr and PQR workerpool to gatsby-worker

a1c7811

chore(gatsby): convert dev-ssr to gatsby-worker

605b723

test(gatsby-cli): one test at a time

6aff32a

pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from ad42d34 to 6aff32a Compare June 28, 2021 11:10

LekoArts reviewed Jun 28, 2021

View reviewed changes

LekoArts suggested changes Jun 29, 2021

View reviewed changes

pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 16aab93 to e48e671 Compare June 29, 2021 09:47

pieh added 3 commits June 29, 2021 11:56

some code shuffling to avoid some error throwing when importing child…

870e71b

… (preperation for messaging API)

drop commented out console.log ( from review comments )

aeca3a6

port debug|inspect cli flag handling from jest-worker

2bf3fac

vladar reviewed Jun 29, 2021

View reviewed changes

pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from c1ff0b5 to 2bf3fac Compare June 29, 2021 09:59

pieh added 2 commits June 29, 2021 12:00

drop skipLibCheck from tsconfig

bd3c1f7

address Vlad's comment

ca74834

pieh force-pushed the pqr/allow-execute-fn-on-all-workers branch from 14741b9 to ca74834 Compare June 29, 2021 10:24

pieh added 4 commits June 29, 2021 12:50

add some jsdocs

7ab0f18

mention why gatsby-worker only expose functions

102007a

document isWorker export

48d001a

add comments about usage of JEST_WORKER_ID which is used by tests, ev…

93bcbbf

…en tho we ourselves will be using gatsby-worker

pieh mentioned this pull request Jun 29, 2021

Provide setup/bootstrap API for workers piscinajs/piscina#136

Closed

LekoArts approved these changes Jun 29, 2021

View reviewed changes

vladar approved these changes Jun 29, 2021

View reviewed changes

pieh merged commit a82f6db into master Jun 29, 2021

pieh deleted the pqr/allow-execute-fn-on-all-workers branch June 29, 2021 13:10

This was referenced Jul 28, 2021

[Snyk] Upgrade gatsby-transformer-sqip from 3.3.1 to 3.9.0 MaxMood96/gatsby#3

Open

[Snyk] Upgrade gatsby-plugin-benchmark-reporting from 0.2.18 to 1.9.0 MaxMood96/gatsby#4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: worker pool that can execute tasks on all workers #32120

feat: worker pool that can execute tasks on all workers #32120

pieh commented Jun 25, 2021 •

edited

LekoArts Jun 28, 2021

LekoArts Jun 29, 2021

pieh Jun 29, 2021

vladar Jun 29, 2021

pieh Jun 29, 2021

vladar left a comment

vladar Jun 29, 2021

pieh Jun 29, 2021

pieh Jun 29, 2021 •

edited

pieh commented Jun 29, 2021

LekoArts left a comment

vladar left a comment

		// if node is not the head then it will have .prev
		taskNode.prev!.next = taskNode.next

feat: worker pool that can execute tasks on all workers #32120

feat: worker pool that can execute tasks on all workers #32120

Conversation

pieh commented Jun 25, 2021 • edited

Description

Perf

Documentation

Related Issues

LekoArts Jun 28, 2021

Choose a reason for hiding this comment

LekoArts Jun 29, 2021

Choose a reason for hiding this comment

pieh Jun 29, 2021

Choose a reason for hiding this comment

vladar Jun 29, 2021

Choose a reason for hiding this comment

pieh Jun 29, 2021

Choose a reason for hiding this comment

vladar left a comment

Choose a reason for hiding this comment

vladar Jun 29, 2021

Choose a reason for hiding this comment

pieh Jun 29, 2021

Choose a reason for hiding this comment

pieh Jun 29, 2021 • edited

Choose a reason for hiding this comment

pieh commented Jun 29, 2021

LekoArts left a comment

Choose a reason for hiding this comment

vladar left a comment

Choose a reason for hiding this comment

pieh commented Jun 25, 2021 •

edited

pieh Jun 29, 2021 •

edited