Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Road to Stabilizing the Standard Library #4600

Open
iuioiua opened this issue Apr 16, 2024 · 16 comments
Open

The Road to Stabilizing the Standard Library #4600

iuioiua opened this issue Apr 16, 2024 · 16 comments
Assignees

Comments

@iuioiua
Copy link
Collaborator

iuioiua commented Apr 16, 2024

The Standard Library (std) aims to provide high-quality, reliable APIs for fearless use in JavaScript and TypeScript projects. For some time, how and when std would reach stability has been in question. Now, thanks to JSR and workspaces functionality in the Deno runtime, we have a path forward.

What stabilization will look like

Currently, std acts as a single module with a single version, meaning a user may be inclined to upgrade, even if a release doesn’t affect the sub-module they use.

After stabilization, new std packages will be exclusively hosted on JSR as separate packages with independent versions. Deno’s workspaces functionality allows us to treat each top-level subdirectory as its own package and better adhere to Semantic Versioning. Dependencies become more manageable by only requiring an upgrade for the packages you use, not the entire std module.

More importantly, independent versioning allows us to work on stabilizing packages asynchronously. We can stabilize more established packages while preparing other packages for stabilization. In other words, we can progressively deliver stable packages for the community sooner rather than later.

These changes will also mean we will release std packages independently of Deno runtime releases.

How we’ll achieve stability

Step 1 - Decouple std from the runtime (complete)

deno and deno_std are now effectively decoupled, so releasing them in tandem is not required. The remaining point of friction is apiland_scripts, which generates docs. We now have two separate GH Actions in apiland_scripts that can be run independently (release CLI and release STD). These actions can be run independently, and it’s a matter of deciding if we do the “split”.

Step 2 - Publish exclusively to JSR (complete)

At the moment, std is published to deno.land/std as-is, then converted into a JSR-compatible form in CI and published to JSR as separate packages under the @std scope. Once the previous step is complete, the codebase can be permanently converted into its JSR-compatible form, providing the granular package versioning for the next step.

Note: For the foreseeable future, deno.land/std will remain online and accessible until the version before the JSR cutover is complete.

Step 3 - Stabilize packages asynchronously (in progress)

Once std is entirely cut over to JSR and packages become more stable, we can begin stabilizing packages. A package will only be stabilized once strict criteria are met that guarantee its quality. These will require that each package is:

  1. Reviewed and approved with sufficient consensus
  2. Tested for maximum coverage
  3. Fully documented and well-designed
  4. Void of any ongoing issues or known bugs
  5. Well supported across many versions of the Deno runtime

Each package follows the stabilization process defined in #4661

Current work

Work towards stabilizing each package is already in progress. This work includes:

You can track the stability status of each package here:

Package Docs Test RC 1.0.0 The issue Stabilization Date RC Planned Date
bytes #4629 Jun 7, 2024
collections #4647 Jun 20, 2024
media-types #4730 Jun 21, 2024
crypto #4885 Jun 30, 2024
encoding #4856 Jun 30, 2024
uuid #4748 Jun 30, 2024
assert #4717 Jul 3, 2024
path #4922 Jul 4, 2024
html #4921 Jul 5, 2024
toml #4923 Jul 6, 2024
url #4924 Jun 7, 2024
data-structures Jun 10, 2024
ulid Jun 11, 2024
regexp Jun 12, 2024
text Jun 13, 2024
msgpack Jun 14, 2024
async Jun 17, 2024
cli Jun 18, 2024
io Jun 19, 2024
streams Jun 20, 2024
fmt Jun 21, 2024
net Jun 24, 2024
http Jun 25, 2024
fs Jun 26, 2024
json Jun 27, 2024
testing Jun 28, 2024
jsonc Jul 1, 2024
csv Jul 2, 2024
semver Jul 3, 2024
expect Jul 4, 2024
yaml Jul 5, 2024
front-matter Jul 8, 2024
ini Jul 9, 2024
dotenv Jul 10, 2024

Excluded Packages

Package Note
datetime Will be re-designed after the Temporal is shipped.
webgpu Excluded because WebGPU is not yet stable.
archive Excluded because it's in the process of migrating from Reader/Writer to Web Stream.
log The design is not ready yet.

If you’d like to contribute to stabilizing the Standard Library in any of this work, please feel free to let us know, either on GitHub or Discord. All contributions are welcome and very much appreciated.

Reference: The dependency graph

The below diagram describes the dependencies between packages. The bottom packages (more depended packages) usually are to be stabilized earlier than the top packages (more depending packages).

@iuioiua iuioiua pinned this issue Apr 16, 2024
@jollytoad
Copy link
Contributor

I'd love to see this become the gold standard of JSR packaging, esp multi-package libraries, and ideally see all scripts/tooling/cross-runtime testing used by the lib become a first class set of modules within the library itself, so that other authors can model their libraries on @std, and benefit from its tooling. I want you follow your model closely in my own library and would be more than happy to contribute to generic tooling and @std itself.

@iuioiua
Copy link
Collaborator Author

iuioiua commented Apr 17, 2024

That'd be great! Improving test coverage is one of the valuable contributions that can be made right now. If you're willing and able, please let us know which package(s) you'd like to look at in #3713. I'll keep you in mind when tooling is needed.

@KnorpelSenf
Copy link

KnorpelSenf commented May 12, 2024

I agree that it makes sense to stabilise the different parts of the standard library at different points in time. However, it is rather annoying to have independent versions for each module.

Currently, we're moving towards a myriad of independent modules, and it adds a fair bit of accidental complexity to let the combinations of (#modules * #versions) explode like that.

I believe that it is a good idea to version the modules independently for the transitional period until almost everything is stable. After that, it would be best to sync up the versions again so that we can return to having The Standard Library again.

@iuioiua
Copy link
Collaborator Author

iuioiua commented May 12, 2024

If all packages share a common, synced version, how should the versions of other packages be affected if a single package needs a release?

so that we can return to having The Standard Library again.

I'm unsure what you mean here. Can you please clarify?

@KnorpelSenf
Copy link

KnorpelSenf commented May 13, 2024

If all packages share a common, synced version, how should the versions of other packages be affected if a single package needs a release?

If there's only a single version, then this version needs to be incremented for every single package. This is fine because a stable standard library should not require a lot of fast fixes, and it is alright if the features are released at a much slower pace. (1-4 times a year maybe?)

so that we can return to having The Standard Library again.

I'm unsure what you mean here. Can you please clarify?

The JavaScript ecosystem is missing a set of useful APIs that are reliable, trustworthy, omnipresent, simple, mature. The modules in this project could fill this exact gap. If they're not aiming to reinvent lodash-but-Deno, but rather trying to be sort of a gold standard (that's what I was trying to imply by calling it “The Standard Library”) then it is important to perceive this as a unit.

So basically, the effect of having independent versions is that std is now scattered into a million pieces, and there's no longer “the thing.”

@jollytoad
Copy link
Contributor

IMHO, the point is that it's a "Library", in this case, a loose collection of packages, which in turn is a collection of modules, it isn't one "thing", ie. it's not "The Standard Package".

Whilst it may seem superficially nice to align the version of each of these packages, the versions will lose all semantic meaning if kept abitrarily in sync. One package version shouldn't be bumped to line up with another if that package hasn't had similar semantic changes, it's just misleading.

For example, in a hypothetical situtation, if I'm working on a pure CLI program I may not care that @std/http has changes, and so unnecessary version bumps to @std/cli would just be annoying.

What I think we definitely need though is for JSR to show the latest versions for each package within a scope... jsr-io/jsr#421

@KnorpelSenf
Copy link

Whilst it may seem superficially nice to align the version of each of these packages, the versions will lose all semantic meaning if kept abitrarily in sync. One package version shouldn't be bumped to line up with another if that package hasn't had similar semantic changes, it's just misleading.

For example, in a hypothetical situtation, if I'm working on a pure CLI program I may not care that @std/http has changes, and so unnecessary version bumps to @std/cli would just be annoying.

I agree, and that's why I was saying that if versions are to be synced up, then the frequent releases have to stop. If you always only get a new standard lib twice a year, then you don't really run into the above problem. The first reason for this is that most packages will have changes of some sort, and the other reason is that even if there's a new release without changes to the parts you use, then this happens so rarely that it doesn't matter much.

The current way of having to do bookkeeping of a large number of modules, each with its own version and stability status, is really annoying, too.

@halvardssm
Copy link
Contributor

I agree with that having the version for the packages in STD in sync would be better for a coherent library, however I also see the argument with it not bringing any benefit in bumping a version without any changes. I am thus not sure if strictly following semantic versioning is the best option for the standard library.

I could see an alternative versioning scheme instead that could potentially work better:

  • patch: any feats or fixes, this version could be updated on a package basis whenever needed and would not be library wide
  • minor: released periodically 1-4 times a year, this would be updated for all packages regardless of updates made to the package
  • major: would signify a major change in the codebase or direction for the library, would be very seldom or potentially never

patch versions would not introduce breaking changes, but minor versions could.

When using packages from the STD, it could be recommended to use ~ for the version range, and manually bump minor versions when needed.

@kt3k
Copy link
Member

kt3k commented May 14, 2024

Yes, we now recommend the users should use version range dependency like @oak/oak@^16.0.0 or @std/path@^0.224.0. These module specifiers will automatically get fixes and new features, if you run the program with --reload option.

Also deno now has deno add <mod> command. This command automatically looks up the latest version of the given jsr package, and add them to deno.json with range dependency set up appropriately. So you even don't need to manage the version number by yourself.

When using packages from the STD, it could be recommended to use ~ for the version range, and manually bump minor versions when needed.

We rather recommend caret ^ range, instead of tilde ~. The caret ^ range includes minor and patch upgrades for >1 versions, and it includes patch upgrades for 0.x versions. This means you'll get fixes and new features automatically when the upstream is upgraded.

@kt3k
Copy link
Member

kt3k commented May 14, 2024

@halvardssm

I could see an alternative versioning scheme instead that could potentially work better:
...
patch versions would not introduce breaking changes, but minor versions could.

This sounds like a significant deviation from semver spec (semver explicitly prohibits the breaking changes in minor upgrades https://semver.org/#summary). I think we should rather follow the semver instead of introducing our own convention.

@KnorpelSenf
Copy link

KnorpelSenf commented May 14, 2024

So you even don't need to manage the version number by yourself.

Maybe not when adding a new module for the first time, but:

  1. you need to update your dependencies at some point, and then you do actually have to figure out the current versions for every module, no matter if ^ did this automatically before
  2. there is no deno update-all-my-jsr-dependencies-to-latest subcommand so people will indeed have to look up the versions manually, or employ some sort of script or other form of automation (renovate etc) that does this
  3. every other major language with a standard library has a single version for it (Rust, Go, even Java and C#, you name it) so making such a mess with many independent versions is breaking expectations

@BlackAsLight
Copy link
Contributor

  1. you need to update your dependencies at some point, and then you do actually have to figure out the current versions for every module, no matter if ^ did this automatically before

Someone did make a script on JSR to update their dependencies. I don't remember what it was called though.

  1. there is no deno update-all-my-jsr-dependencies-to-latest subcommand so people will indeed have to look up the versions manually, or employ some sort of script or other form of automation (renovate etc) that does this

A deno dependency-update would be nice. Maybe someone should suggest that in the Deno repo.

  1. every other major language with a standard library has a single version for it (Rust, Go, even Java and C#, you name it) so making such a mess with many independent versions is breaking expectations

Peoples expectations isn't a reason to do something, and everyone else doing something one way doesn't mean Deno should as well. Tradition for the sake of tradition is a bad reason.

@KnorpelSenf
Copy link

  1. you need to update your dependencies at some point, and then you do actually have to figure out the current versions for every module, no matter if ^ did this automatically before

Someone did make a script on JSR to update their dependencies. I don't remember what it was called though.

There will be a plethora of such scripts if we go ahead with this. Even Node tooling has evolved to update Deno deps: https://www.npmjs.com/package/npm-check-updates

  1. there is no deno update-all-my-jsr-dependencies-to-latest subcommand so people will indeed have to look up the versions manually, or employ some sort of script or other form of automation (renovate etc) that does this

A deno dependency-update would be nice. Maybe someone should suggest that in the Deno repo.

It would be nice, but for other reasons. The point here is “look, the current situation is so complex that we even need new tooling, this is done needlessly” and I hope your takeaway from this isn't “wohoo, we created a superfluous problem, let's solve it by building new tooling!”

  1. every other major language with a standard library has a single version for it (Rust, Go, even Java and C#, you name it) so making such a mess with many independent versions is breaking expectations

Peoples expectations isn't a reason to do something, and everyone else doing something one way doesn't mean Deno should as well.

In fact, building intuitive stuff is a pretty good goal. The opposite of this would be to make things obscure and unexpected, and that surely isn't what we want.

Tradition for the sake of tradition is a bad reason.

You seem to have misunderstood. This is completely unrelated to tradition. The point was not “it's always been this way, let's keep it that way” (then we'd all be using Node) but rather that the rest of the industry isn't stupid. Whenever people innovate a standard library, they conclude that having a single version is good. Doing things differently just for the sake of doing them differently isn't a good approach, either. Instead, we should take a look at why everybody else does it a certain way, and not neglect their findings just because it could be “tradition.”

If we find out that they're indeed all stupid and that having 37+ independent versions is much more simple and intuitive than having 1 version, then sure, we can go on like this :)

@kt3k
Copy link
Member

kt3k commented May 21, 2024

@KnorpelSenf
I think we are still open to the idea of single versioned standard library in some form (such as #4327 . Let's keep this discussion there), but right now we need to split it into pieces to make the stabilization process going forward in a reasonable manner. (See also https://deno.com/blog/std-on-jsr#independent-versioning-for-each-standard-library-package )

Also the Deno community still keeps requesting the breaking changes to the standard library constantly. If we stabilize the entire standard library with a single version, then we probably stop accepting any of such request because the impact of such change would be too large. I personally don't think that situation meets the expectation of the Deno community.

@KnorpelSenf
Copy link

@kt3k awesome, I wasn't aware of #4327, thanks for the reference.

I completely agree that having independent versions is the most reasonable approach as long as things are not stable yet. I tried to make this clear in my initial post already, no disagreement there :)

@roj1512
Copy link
Contributor

roj1512 commented May 21, 2024

I’m also completely agreeing with @kt3k. I’m just worried if not all of the Deno team agrees with that: #4327 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants