D1 beta support

[geelen]

This adds @miniflare/d1 package:

• Statement class, acts similarly to https://github.com/WiseLibs/better-sqlite3/blob/master/docs/api.md#class-statement but is entirely lazy, only truly gets prepared & bound when .all(), .first(), .run() or .raw() are called.
• BetaDatabase class, acts as a D1 binding within Miniflare
• createSQLiteDB returns a better-sqlite3 DB instance for manipulating an in-memory or on-disk Sqlite3 database. Used in Wrangler for running commands against local DBs.
• getSQLiteNativeBindingLocation returns the location of the better_sqlite3.node file, which may not be installed in the current project.

Note this uses npx-import (https://npx-import.pages.dev) to lazily load better-sqlite3, otherwise every installation of miniflare would add ~30s to installation (including npx wrangler ...)

Aside: You can run time npx -y -p better-sqlite3 echo to see how long it takes on your machine:

npx -y -p better-sqlite3@~7.6.2 echo  21.77s user 0.69s system 92% cpu 24.276 total

On a M1 Pro ^

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	fc1d136
Status:	✅  Deploy successful!
Preview URL:	https://bfd3b66b.cf-miniflare.pages.dev
Branch Preview URL:	https://d1-beta-support.cf-miniflare.pages.dev

View logs

[mrbbot]

Nice! 🎉 Added a few comments, mostly around hiding internal methods from users' workers.

[mrbbot]

Closes #277

[CraigglesO]

You may have already fixed this. Browsing the code I couldn't tell. I made a duplicate of this branch for myself just for testing purposes because if you use memory (and possibly persist) in testing:

1. beforeEach function adds data to DB
2. normal test fetches a worker that looks up said data

What happens is the data is wiped because the plugin asks for the DB again... but an in memory DB would just create a new one, and I think sometimes there are issues even with a persist db.

My patch works but isn't great just to show you how I fixed it. But I figured it would be something good to know before this is merged.

[geelen]

That's a good point, @CraigglesO... Each test should start from a clean slate (if you're using in-memory DBs, at least), but beforeEach is really part of a test, so it shouldn't get wiped between.

Does the same problem occur with KV or R2? I wonder if it's something boneheaded I've done with the D1 stuff or a problem more generally...

[CraigglesO]

So KV and R2 pull data in from the storage class.
For instance memory storage just keeps all key-values in a map. BUT, once a storage is created, it's stored in a StorageFactory as a top level plugin in Miniflare, so it won't be dumped as long as the miniflare instance exists (or disposed).

Currently in the case of memory (and I'd argue for persistence as well), it is creating a brand new better-sqlite instance everytime it is fetched.

If you cache the instance for later fetches that should work fine. You can look to DurableObjects for how to cache and cleanup on each reload, and on dispose.

No worries, this project has become somewhat monolithic. I had a terrible first PR but this one looks great otherwise.

@mrbbot - correct me if I'm wrong here please.

[mrbbot]

Looks good! ✅ Couple tiny things then let's get this released 🎉 I'm going to start writing the changelog/docs for 2.9.0 now. 👍

[mrbbot]

BetaDatabase doesn't take a Storage anymore. 🙁

[mrbbot]

BetaDatabase doesn't actually support these functions anymore right?