feat: convert to ESM (breaking change) #856

kanadgupta · 2023-08-09T23:14:14Z

🚥 Partially addresses #801

🧰 Changes

This PR refactors the entire repository over to full ESM. This entailed a few different refactors:

Using .js file extensions for relative imports (really wish I had done this after discovering eslint-plugin-require-extensions) 😭
Updated a bunch of dependencies to their latest ESM counterparts
- (with the exception of ora, see the outstanding issues below)
- Some of our upstream dependencies now use native Node 18 fetch so a test or two had to be updated accordingly. Eventually we'll want to migrate away from nock entirely.
Setting module: NodeNext in our tsconfig.json and type: module in our package.json
In order to build the single-file executable binaries used in our Docker/GitHub Actions builds, we now compile this entire repository into CJS using Rollup before sending it into pkg 😵‍💫 (see ES modules not supported vercel/pkg#1291)
- This obviously isn't ideal, but pkg handles a lot of the madness of compiling into single-file executables better than anything else I've tried (see here for more context)
- A huge benefit here is that our Docker image sizes should be around 25% smaller 📉

Outstanding tasks/issues

Below is a running list of outstanding work — the ora issue is mostly out of our hands, but I'll look into the ExperimentalWarning in a follow-up PR.

Unable to update to the latest version of ora due to segmentation faults at runtime, see ora@7 dependency causes seg fault (was seg fault on mac m1 arm) sindresorhus/ora#229 for more information
Figure out if there's a non-shitty way to hide this warning (edit: I tried fixing this in eeeabd9 but it proved to be problematic for a variety of reasons):

(node:92816) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:92816) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

Unable to properly load in some of our packages
- oas (see fix: get exports working in TS nodeNext oas#796)
- oas-normalize (see fix: get exports working in TS nodeNext oas-normalize#294)
tsconfig cleanup (e.g., do we need downlevelIteration?) (edit: nah let's do this further down the line, we're already introducing a lot of new breaking changes as is)
Move some of these .cjs files over to .js
Unable to build executable for usage in Docker/GitHub Actions

🧬 QA & Testing

Our existing end-to-end tests GitHub Actions tests are passing, meaning this should work, but I'll do some more QA once we tag a beta release.

You can confirm this all works locally by doing the following:

# install/build
npm ci && npm run build:exe

# run the JS output (used by npm-based CLI installations)
bin/rdme.js openapi:validate # or whatever command you want

# run the binary
./exe/rdme openapi:validate  # or whatever command you want

this probably won't work with node 14 at all ugh

most of them are now .cjs with the exception of bin/rdme.js

can we move off jest? all my homies hate jest

kanadgupta · 2023-08-10T14:29:34Z

Update: while I made substantial progress on this, we've hit a pretty massive blocker 😵‍💫 we rely on pkg for our Docker/GitHub Action build, and turns out it doesn't support ESM at all: vercel/pkg#1291

Jon's work in #857 should make this transition a lot easier, but until we figure out a different strategy for building an executable (or if pkg somehow adds ESM support), we probably have to stick with CommonJS for the time being 😭

EDIT (9/2):

Might think about pkg alternatives, like Deno:

https://deno.com/blog/build-cross-platform-cli
https://deno.land/manual@v1.36.4/tools/compiler

Ran into the following error when trying to build an executable with Deno locally:

> deno compile dist/src/cli.js --output deno-test
Compile file:///Users/kanadg/Code/readmeio/rdme/dist/src/cli.js to deno-test
error: Writing deno-test

Caused by:
    0: Reading symlink /Users/kanadg/Code/readmeio/rdme/node_modules/.deno/esbuild@0.18.20/node_modules/@esbuild/win32-arm64
    1: No such file or directory (os error 2)

Deno version info:

deno 1.36.0 (release, aarch64-apple-darwin)
v8 11.6.189.12
typescript 5.1.6

I also tried following Node's guidelines for building a single executable in Node v20.

I haven't troubleshooted this further but I ran into the following error when following the steps above and trying to run the generated executable:

> ./hello
(node:44464) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `hello --trace-warnings ...` to show where the warning was created)
/Users/kanadg/Code/readmeio/rdme/hello:2
import '../dist/src/cli.js';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1153:20)
    at embedderRunCjs (node:internal/util/embedding:18:27)
    at node:internal/main/embedding:18:8

Node.js v20.5.1

Here's my sea-config.json file:

{ "main": "bin/rdme.js", "output": "sea-prep.blob" }

It's weird because running bin/rdme.js works (albeit with a Node warning):

> bin/rdme.js
(node:44508) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:44508) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

                  📖 rdme

    a utility for interacting with ReadMe
       .
       .\                          /.
      ’  ‘                        ‘ ‘
      ( nn\    .           .     /  )
      (nnnnn,.MM.          AM   .nn.
       .nnnndMM----_______--M.’’   /
       |nnn/nnnnnnnnnnnnnnnnn\’mmm/
       /nnnn...nnnnnnnnnnn...mmmmm\
      /nn        ‘qnnnP’       ‘mmm\
      /n’   .XXX. \nnn/   .XX.  \mmb
      An   (XOXX)  nnn   (XOXX)  mm\
     /nn   ‘XXXX’.~”~.   ‘XXX”’ .mmmb
     dnnn.      (    )n.       .mmmmb
    .nnnnnn....n.\ ./nnnnnnnnnmmmmmm\
  (READnnnnnnnnnnn’Y’nnnnnnnnnnmmmmmmME)
  REinnnnnnnnnnnnnnnnnnnnnnnnnmmmmmmmAD/
 /MEEnnnnnnnnnnnnnnnnnnnnnnnnnmmmmmmm)E'.
 READnnnnnnn*’             ‘*mmmmmmmm)MEE.
(READ|nnnn’    \../  \.../    ‘mmmmmM)EEE)
 READ(nn*’                      ‘mmm.MEEE)
  ‘READn’  \._./  \__./  \.../     ‘MEE*’
       *                           /*

Usage

  rdme <command> [arguments] 

Options

  -h, --help       Display this usage guide               
  -v, --version    Show the current rdme version (v8.6.6) 

OpenAPI / Swagger

  $ rdme openapi            Upload, or resync, your OpenAPI/Swagger definition  
                            to ReadMe.                                          
  $ rdme openapi:convert    Convert a Swagger or Postman Collection to OpenAPI. 
  $ rdme openapi:inspect    Analyze an OpenAPI/Swagger definition for various   
                            OpenAPI and ReadMe feature usage.                   
  $ rdme openapi:reduce     Reduce an OpenAPI definition into a smaller subset. 
  $ rdme openapi:validate   Validate your OpenAPI/Swagger definition.           

Docs (a.k.a. Guides)

  $ rdme docs           Sync Markdown files to your ReadMe project as Guides.   
                        Can either be a path to a directory or a single         
                        Markdown file.                                          
  $ rdme docs:prune     Delete any docs from ReadMe if their slugs are not      
                        found in the target folder.                             
  $ rdme guides         Alias for `rdme docs`.                                  
  $ rdme guides:prune   Alias for `rdme docs:prune`.                            

Changelog

  $ rdme changelogs   Sync Markdown files to your ReadMe project as Changelog   
                      posts. Can either be a path to a directory or a single    
                      Markdown file.                                            

Custom Pages

  $ rdme custompages   Sync Markdown/HTML files to your ReadMe project as       
                       Custom Pages. Can either be a path to a directory or a   
                       single Markdown/HTML file.                               

Versions

  $ rdme versions          List versions available in your project or get a     
                           version by SemVer (https://semver.org/).             
  $ rdme versions:create   Create a new version for your project.               
  $ rdme versions:delete   Delete a version associated with your ReadMe         
                           project.                                             
  $ rdme versions:update   Update an existing version for your project.         

Categories

  $ rdme categories          Get all categories in your ReadMe project.         
  $ rdme categories:create   Create a category with the specified title and     
                             guide in your ReadMe project.                      

Administration

  $ rdme login    Login to a ReadMe project.                                    
  $ rdme logout   Logs the currently authenticated user out of ReadMe.          
  $ rdme whoami   Displays the current user and project authenticated with      
                  ReadMe.                                                       

Other useful commands

  $ rdme open   Open your current ReadMe project in the browser. 

Run rdme help <command> for help with a specific command.

To get more help with ReadMe, check out our docs at https://docs.readme.com.

deno needs this

kanadgupta · 2023-09-14T15:49:22Z

src/lib/streamSpecToRegistry.ts

+  formData.append('spec', {
+    type: 'application/json',
+    name: 'openapi.json',
+    [Symbol.toStringTag]: 'File',
+    stream() {
+      return stream;
+    },
+  });


@erunion I'd like your blessing with these changes now that we're using formdata-node — does our API care what we name the file? I just set it to openapi.json but let me know if you feel otherwise.

also our API mocks inspect this payload so we should be in good shape.

kanadgupta · 2023-09-14T15:50:08Z

.github/workflows/ci.yml

-          - 14
-          - 16


with this PR, we're technically requiring Node 18 and above but I'll make that official in a separate PR.

erunion

made it look easy

erunion · 2023-09-14T20:27:31Z

src/lib/streamSpecToRegistry.ts

@@ -24,7 +24,14 @@ export default async function streamSpecToRegistry(spec: string) {

  debug('file and stream created, streaming into form data payload');
  const formData = new FormData();
-  formData.append('spec', stream);
+  formData.append('spec', {


This work seems fine but formdata-node is still pretty shitty. Recommend moving off it and node-fetch as soon as we can. Left a diff in #801 (comment) with the work you can do to this file to move it over to all native APIs.

package.json

rollup.config.js

erunion · 2023-09-14T20:41:31Z

src/lib/readmeAPIFetch.ts

@@ -220,7 +220,9 @@ async function handleRes(res: Response, rejectOnJsonError = true) {
  const contentType = res.headers.get('content-type');
  const extension = mime.extension(contentType);
  if (extension === 'json') {
-    const body = await res.json();
+    // TODO: type this better


Leaving this as any is probably fine imo. You aren't going to have full documented types for these responses without utilizing our API SDK or anything like that (and even then the types are nonexistent for API v1).

yeah node-fetch is remarkably bad about typing responses

# [8.7.0-next.3](v8.7.0-next.2...v8.7.0-next.3) (2023-09-14) ### Features * convert to ESM (breaking change) ([#856](#856)) ([84b8571](84b8571)), closes [/www.stefanjudis.com/snippets/how-to-import-json-files-in-es-modules-node-js/#option-1](https://github.com//www.stefanjudis.com/snippets/how-to-import-json-files-in-es-modules-node-js//issues/option-1) [1#L228](https://github.com/1/issues/L228) [skip ci]

kanadgupta added 14 commits July 6, 2023 11:27

chore: drop module opts in test config

d663e72

Merge branch 'next' into esm-spike

b59744d

feat: the epic renaming

535bf7f

chore(deps): bump chalk to v5

7a14947

chore: lint

6d24836

chore: migrate a few more files that fell through the cracks

89056d4

fix: asserted JSON import

435de45

this probably won't work with node 14 at all ugh

chore: rename config

eba0ba9

chore: rename a bunch of bin files

cf824e3

most of them are now .cjs with the exception of bin/rdme.js

chore: lint

749dc32

chore: try this with jest

b7dbc25

can we move off jest? all my homies hate jest

chore: more jest mess around

f1b1fd4

fix: one more thing in an attempt to fix the build

82ec893

fix: wait is this really all it took

d2b7e51

chore: update npmignore

e3d2ccd

kanadgupta mentioned this pull request Aug 23, 2023

v9 milestone #801

Open

22 tasks

kanadgupta added 10 commits September 2, 2023 18:16

Merge branch 'next' into esm-spike

eaf7824

chore: more extension fixes

365dbc3

ci: knip fix

b4ac068

refactor: enable node protocol

829170c

deno needs this

Merge branch 'next' into esm-spike

2f4a29d

chore: lint

b12f384

chore: fix up some more extensions

2f23937

revert: rename file again

35e1199

fix: another thing that fell through the cracks

2b8cfc6

chore: one more fix

6927124

kanadgupta changed the title ~~ESM spike~~ refactor: esm Sep 7, 2023

kanadgupta added 2 commits September 11, 2023 13:57

chore: bring in new eslint rule, fix up some more issues

b09e6e1

Merge branch 'next' into esm-spike

aa22557

kanadgupta added 3 commits September 13, 2023 10:57

chore: fix up ignore files again

827d1fd

chore: comment

380ca23

chore: one of these days i'll be content with my ignore file formatting

07ba2f6

kanadgupta commented Sep 14, 2023

View reviewed changes

kanadgupta marked this pull request as ready for review September 14, 2023 15:57

kanadgupta changed the title ~~refactor: esm~~ feat: convert to ESM (breaking change) Sep 14, 2023

kanadgupta requested review from erunion, a team, Dashron, darrenyong, RyanGWU82, domharrington, rafegoldberg and ilias-t and removed request for a team September 14, 2023 15:59

kanadgupta added 2 commits September 14, 2023 11:05

Merge branch 'next' into esm-spike

a6ba541

chore(deps): bump headers-polyfill

43e0a66

kanadgupta requested a review from llimllib September 14, 2023 16:10

kanadgupta added 3 commits September 14, 2023 12:22

refactor: hide rollup output in dist

bc99152

revert: remove unused debug flag

e83b2e6

Merge branch 'next' into esm-spike

4827139

erunion approved these changes Sep 14, 2023

View reviewed changes

kanadgupta added 4 commits September 14, 2023 16:13

chore: swap out headers-polyfill type for node-fetch

5a2a9e3

Merge branch 'next' into esm-spike

4a4c852

chore: lint

d758dd5

fix: type fixes

13fa09f

kanadgupta merged commit 84b8571 into next Sep 14, 2023
8 checks passed

kanadgupta deleted the esm-spike branch September 14, 2023 21:52

kanadgupta added this to the v9 milestone Sep 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: convert to ESM (breaking change) #856

feat: convert to ESM (breaking change) #856

kanadgupta commented Aug 9, 2023 •

edited

kanadgupta commented Aug 10, 2023 •

edited

kanadgupta Sep 14, 2023 •

edited

kanadgupta Sep 14, 2023

erunion left a comment

erunion Sep 14, 2023

erunion Sep 14, 2023

kanadgupta Sep 14, 2023

feat: convert to ESM (breaking change) #856

feat: convert to ESM (breaking change) #856

Conversation

kanadgupta commented Aug 9, 2023 • edited

🧰 Changes

🧬 QA & Testing

kanadgupta commented Aug 10, 2023 • edited

kanadgupta Sep 14, 2023 • edited

Choose a reason for hiding this comment

kanadgupta Sep 14, 2023

Choose a reason for hiding this comment

erunion left a comment

Choose a reason for hiding this comment

erunion Sep 14, 2023

Choose a reason for hiding this comment

erunion Sep 14, 2023

Choose a reason for hiding this comment

kanadgupta Sep 14, 2023

Choose a reason for hiding this comment

kanadgupta commented Aug 9, 2023 •

edited

kanadgupta commented Aug 10, 2023 •

edited

kanadgupta Sep 14, 2023 •

edited