Re-evaluate usage of Jest as test-runner

[akphi]

jest does not fully support ESM - See jestjs/jest#9430. When we upgrade and use ESM support, we can:

• Get rid of the main field in package.json and let babel-jest handle the transform for these ESM modules - There is currently a problem with this - Support package exports in jest-resolve jestjs/jest#9771
• Get rid of custom logic to handle ESM for lodash-es - Merge lodash-es into lodash package lodash/lodash#5107
• uuid does not fit well with jest-resolve - fix: add browser condition to package.json uuidjs/uuid#616
• We cannot yet just go full out ESM because of all the problems in - Meta: Native support for ES Modules jestjs/jest#9430 (comment) - mainly to do with usage of jest.mock and jest.requiresActual which are only used in GrammarRoundtrip tests right now. We will be able to get rid of these by using process env variable hack (e.g. process.env.TEST_GRAMMAR_ROUNDTRIP)

---

This is extracted from the umbrella tracker for dependencies #83
We will close this once we can fully adopt ESM in our usage with Jest and make necessary adjustments in the codebase

---

UPDATED:

We have adopted ESM exports style, but we haven't fully immerse in ESM mode yet as it seems from Node side there are quite a number of stabilization to be done. And holes to patch, such as problems with uuid, lodash-es as mentioned above, so I will keep this PR open for now.

[akphi]

I also tried to use @jest/global - https://jestjs.io/blog/2020/05/05/jest-26#a-new-way-to-consume-jest---jestglobals - I have tried to make it work with the following patch

0001-use-jest-globals.patch.txt

but there are a few problems:

• There are quite a handful number of type hacks to be done
• There are problems with jest-extended [Feature]: Replace @types/jest's expect implementation with @jest/expect jestjs/jest#12424
• When injectGlobals=false, some test seem to be failing as they assume the global jest i.e. jest-canvas-mock, testing-library`, etc. so maybe it's not worth the effort?

[akphi]

I have updated the codebase to use @jest/globals but still stuck with injectGlobals. Seems like that flag has more implications than just removing the globals, I start seeing errors in testing-library and jsdom as well.

See #1190

[akphi]

Workaround regarding issue with uuid has been resolved by #1453

[akphi]

Jest has this weird problem around Promise, sometimes, .mockResolvedValue() does not work as intended
See jestjs/jest#2549
See jestjs/jest#6645
See jestjs/jest#2549

Maybe it's not a bad idea to switch to another test framework, we could consider https://github.com/vitest-dev/vitest
Looks like it has a killer VSCode plugin for running tests https://marketplace.visualstudio.com/items?itemName=ZixuanChen.vitest-explorer
Read more about Vite vs Jest here - https://blog.bitsrc.io/vite-is-better-than-webpack-d5dd59610d56

There seems to be a small problem with its performance
vitest-dev/vitest#579
https://vitest.dev/guide/features.html#threads
https://blog.logrocket.com/testing-vite-minimal-config-using-vitest/

---

on this note, we could also look at Vite instead of webpack to simplify toolchain