add types for iterator helpers proposal #58222

bakkot · 2024-04-17T02:59:15Z

I have only minimal tests because almost all the types are copied directly from Array with very minimal changes (mostly dropping the third "array" parameter to callbacks and the "thisArg" parameter to callback-taking methods, and returning NativeIterator<T> instead of T[]). But I'm happy to make the tests more extensive if you'd like.

For reviewing:

ignore bd51b00 through 30d9bc2; the last one reverts the earlier ones so the next commits can be clearer
f5393db and ebc4e71 add the new BuiltinIterator and AsyncBuiltinIterator types and adopt them everywhere except the .generated files. AsyncBuiltinIterator isn't necessary yet but I figured we might as well do this plumbing work now.
c4a8a08 adds the new types following the strategy @rbuckton suggested below

The "rebaseline" commits are just running npm run test -- --no-lint; npx hereby baseline-accept and committing the result.

Current status (as of 2024-04-25):

waiting on Add TReturn/TNext to Iterable et al #58243
needs CloseableBuiltinIterator or something to mean "built-in iterator with a return method"
needs a rebase
needs reviews from the team

I'll address the outstanding todos one #58243 lands.

fatcerberus · 2024-04-17T03:25:54Z

I will need someone to tell me how to handle the generated DOM typings

The DOM typings are maintained at https://github.com/microsoft/TypeScript-DOM-lib-generator

bakkot · 2024-04-17T03:39:32Z

More concretely, I'll need someone to tell me how to handle the change: changing the DOM typings to use NativeIterator would break anyone not importing esnext.iterator, but there's not (currently) an equivalent of esnext for DOM typings, so there's not anywhere else to do the update.

Renegade334 · 2024-04-17T10:21:06Z

More concretely, I'll need someone to tell me how to handle the change: changing the DOM typings to use NativeIterator would break anyone not importing esnext.iterator, but there's not (currently) an equivalent of esnext for DOM typings, so there's not anywhere else to do the update.

Would the following work?

Introduce a hollow NativeIterator interface in es2015.iterable.d.ts
Re-type all intrinsic appearances of IterableIterator in es2015.iterable.d.ts etc. to NativeIterator
Extend the NativeIterator interface in esnext.iterator.d.ts, add the constructor definition, etc.

This would allow what you're looking for: keeps compatibility for pre-esnext targets, while allowing the helper methods to be merged into NativeIterator in esnext. The DOM library could then be retyped without causing BC issues.

Sparse illustration:

// @target: esnext
// @lib: esnext

// @filename: lib.es2015.iterable.d.ts

interface Iterator<T, TReturn = any, TNext = undefined> {
    next(...args: [] | [TNext]): IteratorResult<T, TReturn>;
    return?(value?: TReturn): IteratorResult<T, TReturn>;
    throw?(e?: any): IteratorResult<T, TReturn>;
}

interface Iterable<T> {
    [Symbol.iterator](): Iterator<T>;
}

interface IterableIterator<T> extends Iterator<T> {
    [Symbol.iterator](): IterableIterator<T>;
}

interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {
    [Symbol.iterator](): NativeIterator<T, TReturn, TNext>;
}

interface Generator<T = unknown, TReturn = any, TNext = unknown> extends NativeIterator<T, TReturn, TNext> {
    // etc.
}

interface Array<T> /* etc. */ {
    // was: [Symbol.iterator](): IterableIterator<T>;
    [Symbol.iterator](): NativeIterator<T>;

    // was: entries(): IterableIterator<[number, T]>;
    entries(): NativeIterator<[number, T]>;

    // etc.
}


// @filename: lib.esnext.iterator.d.ts

interface NativeIterator<T, TReturn, TNext> {
    map<U>(callbackfn: (value: T, index: number) => U): NativeIterator<U>;

    // etc.
}

// @filename: test.ts

const mappedArrayIterator = ['a', 'b', 'c'].entries().map(([k, s]) => k * s.charCodeAt(0)); // NativeIterator<number>

const castToIterableIterator: IterableIterator<number> = mappedArrayIterator; // NativeIterator<T> remains assignable to IterableIterator<T>, for what it's worth

As for the question of extending IterableIterator versus creating a new interface for native Iterator instances, my own $0.02 is in favour of the latter. IterableIterator is a mature interface, and it isn't just used internally: a cursory glance through GH shows quite a few examples of manually-implemented iterable iterator objects in the ecosystem, and these remain valid iterables/iterators in esnext even though they are not descended from Iterator.prototype and don't implement helper methods. Maintaining separation between the "implements iterator protocols" interface and the "descends from Iterator.prototype" interface makes sense logically, and has the added benefit of not breaking third-party implementations of the existing interface.

bakkot · 2024-04-17T15:01:31Z

Would the following not work?

Nice, that sounds like a good approach to me.

rbuckton · 2024-04-17T16:44:54Z

@bakkot the approach I am experimenting with to support abstract on NativeIterator looks something like this:

// lib.esnext.iterator.d.ts
/// <reference lib="es2015.iterable" />
export {};

// Abstract type that allows us to mark `next` as `abstract`
declare abstract class Iterator<T> {
  abstract next(value?: undefined): IteratorResult<T, void>;
}

// Merge all members of `NativeIterator<T>` into `Iterator<T>`
interface Iterator<T> extends globalThis.NativeIterator<T, void, undefined> {}

// Capture the `Iterator` constructor in a type we can use in the `extends` clause of `IteratorConstructor`.
type NativeIteratorConstructor = typeof Iterator;

declare global {
  // Global `NativeIterator<T>` interface that can be augmented by polyfills
  interface NativeIterator<T, TReturn, TNext> {
    // prototype elements
  }

  // Global `IteratorConstructor` interface that can be augmented by polyfills
  interface IteratorConstructor extends NativeIteratorConstructor {
    // static elements
  }

  var Iterator: IteratorConstructor;
}

// lib.es2015.iterable.d.ts

...

interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {
  [Symbol.iterator](): NativeIterator<T>;
}

...

And in use:

new Iterator<number>(); // ts(2511): Cannot create an instance of an abstract class.

class C extends Iterator<number> {} // ts(2515): Non-abstract class 'C' does not implement inherited
                                    //           abstract member next from class 'Iterator<number>'.

Lib references can't really be referenced as modules, and even if you could this provides no exports. However, we will still augment the global scope with the types defined in declare global.

rbuckton · 2024-04-17T16:48:19Z

There need to be a few other changes so that it can also be used with generators, though.

rbuckton · 2024-04-17T16:59:11Z

src/lib/esnext.iterator.d.ts

+    find<S extends T>(predicate: (value: T, index: number) => value is S): S | undefined;
+    find(predicate: (value: T, index: number) => unknown): T | undefined;
+
+    readonly [Symbol.toStringTag]: "Iterator";


I would suggest defining this as

Suggested change

readonly [Symbol.toStringTag]: "Iterator";

readonly [Symbol.toStringTag]: string;

otherwise subclasses of Iterator won't be able to redefine it.

rbuckton · 2024-04-17T17:14:17Z

src/lib/esnext.iterator.d.ts

+
+declare var Iterator: (abstract new <T>() => NativeIterator<T>) & IteratorConstructor;
+
+// TODO BEFORE MERGING: update all existing IterableIterator-return methods to return NativeIterator


As suggested elsewhere, I would suggest you define NativeIterator as follows in the es2015 libs and then just update all of the IterableIterator references to NativeIterator:

interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> { [Symbol.iterator](): NativeIterator<T, TReturn, TNext>; }

Should I split that into a separate PR, or do it here?

rbuckton · 2024-04-17T17:21:11Z

src/lib/esnext.iterator.d.ts

@@ -0,0 +1,133 @@
+interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {


The export {} trick mentioned in my earlier comment can address the abstract next() method definition.

Jack-Works · 2024-04-17T17:44:24Z

src/lib/esnext.iterator.d.ts

@@ -0,0 +1,133 @@
+interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {


The word choose "Native" is not so good. Do we have other options?

@rbuckton had previously suggested Builtin, which I'm also fine with.

Renegade334 · 2024-04-17T17:44:38Z

@rbuckton does the interface merging make any existing "custom" Iterator objects fail type validation in target:esnext, since they won't implement the new Iterator.prototype methods?

Edit: I overlooked the module hacking. Very nice.

Jack-Works · 2024-04-17T17:45:44Z

@rbuckton does the interface merging make any existing "custom" Iterator objects fail type validation in target:esnext, since they won't implement the new Iterator.prototype methods?

It does not merging the current Iterator interface, but create a new one called NativeIterator

rbuckton · 2024-04-17T20:28:49Z

@rbuckton does the interface merging make any existing "custom" Iterator objects fail type validation in target:esnext, since they won't implement the new Iterator.prototype methods?

@bakkot is correct. This should not affect anyone implementing a custom Iterator, or even IterableIterator.

After this is in we may want to add a shortcuts for it to getIterationTypesOfIterableFast and getIterationTypesOfIteratorFast in the checker.

This reverts commit bd51b00. (and subsequent commits)

…es to that

bakkot · 2024-04-18T02:04:00Z

Updated almost exactly following this comment, except that I had BuiltinIterator's Symbol.iterator method pass all the type arguments to the return type; this is necessary so Generator can extend BuiltinIterator (otherwise the Symbol.iterator methods' types conflict).

A consequence of this change is that the type of yield* [] is now void instead of any, because the default for TReturn on BuiltinIterator is void instead of any (as it is for `Iterator).

Also, passing void for the second argument in the return type in next in

declare abstract class Iterator<T> {
    abstract next(value?: undefined): IteratorResult<T, void>;
}

means that classes attempted to implement Iterator have some pain: you can't just return { done: false, value: 0 } from your next because the done value there will be inferred as boolean, which means that the value field has to conform to void so that it can extend the { done: true, value: TReturn } half of IteratorResult<T, void>, which it does not. Also, you can't make an iterator which actually does have a non-void value for the done: true case, which seems like an unnecessary constraint.

I have test cases illustrating the difficulties:

    class BadIterator1 extends Iterator<number> {
      next() {
      ~~~~
!!! error TS2416: Property 'next' in type 'BadIterator1' is not assignable to the same property in base type 'Iterator<number>'.
!!! error TS2416:   Type '() => { readonly done: false; readonly value: 0; } | { readonly done: true; readonly value: "a string"; }' is not assignable to type '(value?: undefined) => IteratorResult<number, void>'.
!!! error TS2416:     Type '{ readonly done: false; readonly value: 0; } | { readonly done: true; readonly value: "a string"; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:       Type '{ readonly done: true; readonly value: "a string"; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:         Type '{ readonly done: true; readonly value: "a string"; }' is not assignable to type 'IteratorReturnResult<void>'.
!!! error TS2416:           Types of property 'value' are incompatible.
!!! error TS2416:             Type 'string' is not assignable to type 'void'.
        if (Math.random() < .5) {
          return { done: false, value: 0 } as const;
        } else {
          return { done: true, value: "a string" } as const;
        }
      }
    }
    
    class BadIterator2 extends Iterator<number> {
      next() {
      ~~~~
!!! error TS2416: Property 'next' in type 'BadIterator2' is not assignable to the same property in base type 'Iterator<number>'.
!!! error TS2416:   Type '() => { done: boolean; value: number; }' is not assignable to type '(value?: undefined) => IteratorResult<number, void>'.
!!! error TS2416:     Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:       Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorReturnResult<void>'.
!!! error TS2416:         Types of property 'done' are incompatible.
!!! error TS2416:           Type 'boolean' is not assignable to type 'true'.
        return { done: false, value: 0 };
      }
    }
    
    class BadIterator3 extends Iterator<number> {
      next() {
      ~~~~
!!! error TS2416: Property 'next' in type 'BadIterator3' is not assignable to the same property in base type 'Iterator<number>'.
!!! error TS2416:   Type '() => { done: boolean; value: number; } | { done: boolean; value: string; }' is not assignable to type '(value?: undefined) => IteratorResult<number, void>'.
!!! error TS2416:     Type '{ done: boolean; value: number; } | { done: boolean; value: string; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:       Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorResult<number, void>'.
!!! error TS2416:         Type '{ done: boolean; value: number; }' is not assignable to type 'IteratorReturnResult<void>'.
!!! error TS2416:           Types of property 'done' are incompatible.
!!! error TS2416:             Type 'boolean' is not assignable to type 'true'.
        if (Math.random() < .5) {
          return { done: false, value: 0 };
        } else {
          return { done: true, value: "a string" };
        }
      }
    }

I see a few approaches to improving this last issue:

pass any as the second parameter for IteratorResult in the abstract class Iterator (this fixes the BadIterator1 and BadIterator2 cases)
pass unknown as the second parameter for IteratorResult in the above (this fixes the BadIterator1 and BadIterator2 cases)
make abstract class Iterator take a second parameter for TReturn, defaulting to void (this fixes the BadIterator1 and BadIterator2 cases if the user explicitly specifies the second parameter in the classes)

bakkot · 2024-04-18T02:10:08Z

One detail not captured by the types here is that the iterator returned by helpers like map unconditionally has a return method. I guess that's probably worth expressing?

rbuckton · 2024-04-18T16:56:38Z

means that classes attempted to implement Iterator have some pain: you can't just return { done: false, value: 0 } from your next because the done value there will be inferred as boolean, which means that the value field has to conform to void so that it can extend the { done: true, value: TReturn } half of IteratorResult<T, void>, which it does not.

We could probably add TReturn = void, TNext = undefined type parameters to the built in Iterator class without issue, since it's up to the user to implement next anyways.

rbuckton · 2024-04-19T14:04:31Z

One detail not captured by the types here is that the iterator returned by helpers like map unconditionally has a return method. I guess that's probably worth expressing?

Do helpers like map also have a throw method? If so, you could just use Generator for those. If not, we'd have to add another NativeIterator subtype for those.

rbuckton · 2024-04-19T15:29:29Z

Regarding the typing issues, I'm hoping #58243 might help with that.

bakkot · 2024-04-19T15:41:31Z

Do helpers like map also have a throw method? If so, you could just use Generator for those. If not, we'd have to add another NativeIterator subtype for those.

Nope, just return. throw is really a generator-specific thing.

Suggestions for the name? CloseableBuiltinIterator, maybe, since presence of return indicates that you can close the iterator?

rbuckton · 2024-04-19T18:28:00Z

Suggestions for the name? CloseableBuiltinIterator, maybe, since presence of return indicates that you can close the iterator?

I suppose that's fine. @DanielRosenwasser, do you have any thoughts on the name for this type?

felixfbecker · 2024-04-24T22:01:56Z

Some more ideas:

HelpfulIterator
HelperIterator
TransformableIterator
IteratorInstance

Josh-Cena · 2024-04-25T15:49:50Z

This "new class extends Iterator does not return something cross-compatible with Iterator" behavior still sounds confusing to me, IMO.

bakkot · 2024-04-25T16:04:33Z

@Josh-Cena Not sure what you mean. It is cross-compatible as long as your class implements the next method, which you are required to do anyway because you're extending an abstract class. That is, this works:

class GoodIterator extends Iterator<number> {
  next() {
    return { done: false, value: 0 } as const;
  }
}

let x: Iterator<number> = new GoodIterator();

What's a concrete example of the case you're worried about?

Josh-Cena · 2024-04-25T16:40:49Z

Sorry, I didn't say what I meant to say.

If we have the following:

declare const x: Iterator<number>;

const y = new Iterator();

Then we expect x and y to be mutually replaceable with each other. Now because here Iterator is an abstract class, new Iterator() doesn't make sense, so let's use your GoodIterator instead—but, there should be no difference because GoodIterator merely overrides an abstract method but does not implement any additional behavior. However, I would be surprised to discover that x is not iterable, despite the fact that the abstract class Iterator provides Symbol.iterator.

declare const x: Iterator;

for (const val of x) {
  // Doesn't work
}

It's like a value of type Map and new Map doing different things.

Josh-Cena · 2024-04-25T16:43:08Z

Actually, I see that in esnext Iterator would also have the methods of BuiltInIterator, I'm still slightly surprised as we didn't really "add" Symbol.iterator in esnext...

bakkot · 2024-04-25T16:52:27Z

Ah, yeah. That's an unavoidable consequence of TS already having an Iterator interface, though - the only alternative is to change meaning of the the type-level Iterator, which would be a pretty large breaking change. I think this ends up being the best available option, despite the downside you point out.

Josh-Cena · 2024-04-25T17:11:33Z

Um... I see the reason. There still are breaking changes and anyone using esnext might not be able to interop with old libraries, though. Instead of adding a new name which is virtually impossible to remove in the future (while its utility diminishes as people adopt the current esnext), I still feel it's better to accept a minor breaking change to Iterator, assuming people nearly always use generators anyway.

bakkot · 2024-04-25T17:28:09Z

There still needs to be a distinction between "has a next method" and "has all the methods given by the built-in ES Iterator class", so we still need to have two named types - we can't just add more stuff to the existing TS Iterator interface and be done with it. I think it makes sense for the first to be named "Iterator" and the second to be named BuiltinIterator.

Renegade334 · 2024-04-25T23:05:08Z

Um... I see the reason. There still are breaking changes and anyone using esnext might not be able to interop with old libraries, though. Instead of adding a new name which is virtually impossible to remove in the future (while its utility diminishes as people adopt the current esnext), I still feel it's better to accept a minor breaking change to Iterator, assuming people nearly always use generators anyway.

There is no breaking change; in fact, quite the opposite. The declarations in esnext.iterators.d.ts are defined under module scope, and then selectively exported into the global namespace in such a way that they don't interfere with the existing Iterator interface.

Referencing Iterator by value will refer to the new global Iterator constructor. Referencing Iterator by type will refer to the existing "an object that implements iterator methods" interface. The former allows users to refer to Iterator.prototype and extend the class if desired, while the latter ensures backward compatibility.

Josh-Cena · 2024-04-26T04:09:14Z

I missed the export {} in the iterator def. Then my only qualm is the divergence between value-space and type-space Iterator structure.

bakkot added 2 commits April 16, 2024 19:46

add types for iterator helpers proposal

bd51b00

run baselines

075ce43

typescript-bot added the For Uncommitted Bug PR for untriaged, rejected, closed or missing bug label Apr 17, 2024

format

4449c82

bakkot mentioned this pull request Apr 17, 2024

PSA: potential lib breaking change after iterator-helpers proposal #54481

Open

rbuckton requested changes Apr 17, 2024

View reviewed changes

Jack-Works reviewed Apr 17, 2024

View reviewed changes

bakkot added 6 commits April 17, 2024 15:55

Revert "add types for iterator helpers proposal"

30d9bc2

This reverts commit bd51b00. (and subsequent commits)

add BuiltinIterator and switch existing IterableIterator uses to that

f5393db

add AsyncBuiltinIterator and switch existing AsyncIterableIterator us…

ebc4e71

…es to that

rebaseline

e748158

add types for iterator helpers

c4a8a08

rebaseline

51124ae

rbuckton mentioned this pull request Apr 18, 2024

Add TReturn/TNext to Iterable et al #58243

Open

rbuckton mentioned this pull request May 1, 2024

Add synthetic TypeScriptSettings interface that exposes some compiler options to type system #58396

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add types for iterator helpers proposal #58222

add types for iterator helpers proposal #58222

bakkot commented Apr 17, 2024 •

edited

fatcerberus commented Apr 17, 2024

bakkot commented Apr 17, 2024

Renegade334 commented Apr 17, 2024 •

edited

bakkot commented Apr 17, 2024

rbuckton commented Apr 17, 2024 •

edited

rbuckton commented Apr 17, 2024

rbuckton Apr 17, 2024

rbuckton Apr 17, 2024

bakkot Apr 17, 2024

rbuckton Apr 17, 2024

Jack-Works Apr 17, 2024

bakkot Apr 17, 2024

Renegade334 commented Apr 17, 2024 •

edited

Jack-Works commented Apr 17, 2024

rbuckton commented Apr 17, 2024

bakkot commented Apr 18, 2024

bakkot commented Apr 18, 2024

rbuckton commented Apr 18, 2024

rbuckton commented Apr 19, 2024

rbuckton commented Apr 19, 2024

bakkot commented Apr 19, 2024

rbuckton commented Apr 19, 2024

felixfbecker commented Apr 24, 2024

Josh-Cena commented Apr 25, 2024

bakkot commented Apr 25, 2024

Josh-Cena commented Apr 25, 2024 •

edited

Josh-Cena commented Apr 25, 2024

bakkot commented Apr 25, 2024

Josh-Cena commented Apr 25, 2024 •

edited

bakkot commented Apr 25, 2024

Renegade334 commented Apr 25, 2024

Josh-Cena commented Apr 26, 2024

	readonly [Symbol.toStringTag]: "Iterator";
	readonly [Symbol.toStringTag]: string;


		declare var Iterator: (abstract new <T>() => NativeIterator<T>) & IteratorConstructor;

		// TODO BEFORE MERGING: update all existing IterableIterator-return methods to return NativeIterator

		@@ -0,0 +1,133 @@
		interface NativeIterator<T, TReturn = void, TNext = undefined> extends Iterator<T, TReturn, TNext> {

add types for iterator helpers proposal #58222

Are you sure you want to change the base?

add types for iterator helpers proposal #58222

Conversation

bakkot commented Apr 17, 2024 • edited

fatcerberus commented Apr 17, 2024

bakkot commented Apr 17, 2024

Renegade334 commented Apr 17, 2024 • edited

bakkot commented Apr 17, 2024

rbuckton commented Apr 17, 2024 • edited

rbuckton commented Apr 17, 2024

rbuckton Apr 17, 2024

Choose a reason for hiding this comment

rbuckton Apr 17, 2024

Choose a reason for hiding this comment

bakkot Apr 17, 2024

Choose a reason for hiding this comment

rbuckton Apr 17, 2024

Choose a reason for hiding this comment

Jack-Works Apr 17, 2024

Choose a reason for hiding this comment

bakkot Apr 17, 2024

Choose a reason for hiding this comment

Renegade334 commented Apr 17, 2024 • edited

Jack-Works commented Apr 17, 2024

rbuckton commented Apr 17, 2024

bakkot commented Apr 18, 2024

bakkot commented Apr 18, 2024

rbuckton commented Apr 18, 2024

rbuckton commented Apr 19, 2024

rbuckton commented Apr 19, 2024

bakkot commented Apr 19, 2024

rbuckton commented Apr 19, 2024

felixfbecker commented Apr 24, 2024

Josh-Cena commented Apr 25, 2024

bakkot commented Apr 25, 2024

Josh-Cena commented Apr 25, 2024 • edited

Josh-Cena commented Apr 25, 2024

bakkot commented Apr 25, 2024

Josh-Cena commented Apr 25, 2024 • edited

bakkot commented Apr 25, 2024

Renegade334 commented Apr 25, 2024

Josh-Cena commented Apr 26, 2024

bakkot commented Apr 17, 2024 •

edited

Renegade334 commented Apr 17, 2024 •

edited

rbuckton commented Apr 17, 2024 •

edited

Renegade334 commented Apr 17, 2024 •

edited

Josh-Cena commented Apr 25, 2024 •

edited

Josh-Cena commented Apr 25, 2024 •

edited