Discussion: Hidden tuples to track Observable size and type

[kolodny]

I'd like to suggest RxJS using hidden tuples to have stronger types which will also catch some logic errors at compile time

The general idea is that an Observable will have an internal tuple type of the elements of the Observable in situations when it's known. For example a call to
of(1, 2, 'test') will produce Observable<number|string, [number, number, string]>
while at the same time
let els = [1, 2, 'test']; of(...els) will produce
Observable<number|string, (number|string)[]>

This will allow things like:

of(1, 2, 3, 'foo', 'bar', false).pipe(take(2)).subscribe((thisIsOnlyANumber) => {
  // thisIsOnlyANumber is number and not number | string | boolean
})

It also empowers us to disallow invalid operators invocations like

of(1, 2).pipe(take(3))
                   ~ Argument of type '3' is not assignable to parameter of type '1 | 2'.

This technique would also give us a Single for free since fromFetch(), from(promise), and generated API could be set this hidden tuple to a single element. This can also remove the need for folks to do things like api.fetchData().pipe(take(1)).

I haven't quite worked out the correct typings yet but here's a rough approximation of what this would look like

type FromArray<T extends any[]> = T extends Array<infer U> ? U : never;

declare function of<T extends any[]>(...t: T): Observable<FromArray<T>, T>;

interface MonoTypeOperatorFunction<T, Elements extends T[] = T[]> extends OperatorFunction<T, T, Elements> {}
interface UnaryFunction<T, R> { (source: T): R; }
interface OperatorFunction<T, R, Elements extends T[] = T[]> extends UnaryFunction<Observable<T, Elements>, Observable<R>> {}

class Observable<T, Elements extends T[] = T[]> {
  subscribe(callback: (t: T) => void) {}

  pipe(): Observable<T>;
  pipe<A, AS extends A[]>(op1: OperatorFunction<T, A, Elements>): Observable<A, AS>;
  pipe<A, AS extends A[], B, BS extends B[]>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B, AS>): Observable<B>;
  pipe<A, AS extends A[], B, C>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>): Observable<C>;
  pipe<A, AS extends A[], B, C, D>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>, op4: OperatorFunction<C, D>): Observable<D>;
  pipe<A, AS extends A[], B, C, D, E>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>, op4: OperatorFunction<C, D>, op5: OperatorFunction<D, E>): Observable<E>;
  pipe<A, AS extends A[], B, C, D, E, F>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>, op4: OperatorFunction<C, D>, op5: OperatorFunction<D, E>, op6: OperatorFunction<E, F>): Observable<F>;
  pipe<A, AS extends A[], B, C, D, E, F, G>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>, op4: OperatorFunction<C, D>, op5: OperatorFunction<D, E>, op6: OperatorFunction<E, F>, op7: OperatorFunction<F, G>): Observable<G>;
  pipe<A, AS extends A[], B, C, D, E, F, G, H>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>, op4: OperatorFunction<C, D>, op5: OperatorFunction<D, E>, op6: OperatorFunction<E, F>, op7: OperatorFunction<F, G>, op8: OperatorFunction<G, H>): Observable<H>;
  pipe<A, AS extends A[], B, C, D, E, F, G, H, I>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>, op4: OperatorFunction<C, D>, op5: OperatorFunction<D, E>, op6: OperatorFunction<E, F>, op7: OperatorFunction<F, G>, op8: OperatorFunction<G, H>, op9: OperatorFunction<H, I>): Observable<I>;
  pipe<A, AS extends A[], B, C, D, E, F, G, H, I>(op1: OperatorFunction<T, A, Elements>, op2: OperatorFunction<A, B>, op3: OperatorFunction<B, C>, op4: OperatorFunction<C, D>, op5: OperatorFunction<D, E>, op6: OperatorFunction<E, F>, op7: OperatorFunction<F, G>, op8: OperatorFunction<G, H>, op9: OperatorFunction<H, I>, ...operations: OperatorFunction<any, any>[]): Observable<unknown>;
  pipe(...operations: OperatorFunction<any, any>[]): Observable<any> {
    return {} as any;
  }
  
}

function take<T, Elements extends T[] = T[]>(
  count: Elements extends [] ? never :
    Elements extends [any] ? 1 :
    Elements extends [any, any] ? 1|2 :
    Elements extends [any, any, any] ? 1|2|3 :
    Elements extends [any, any, any, any] ? 1|2|3|4 :
    Elements extends [any, any, any, any, any] ? 1|2|3|4|5 :
    Elements extends any[] ? number :
    never
): OperatorFunction<T, T, Elements> {
  return (source: Observable<T, Elements>) => {
    return {} as any;
  };
}

of(1, 2).pipe(take(2)) // OK
of(1, 2).pipe(take(4)) // Error: Argument of type '4' is not assignable to parameter of type '1 | 2'.

let a$: Observable<string | number> = of(1, 'test');
a$.pipe(take(10)); // OK since the hidden tuple information was discarded from the explicit typing.

The typings can get complex pretty easily, for example I haven't been able to get of(1, '', boolean).pipe(take(2)) to produce an Observable<number | string> since that needs serious tuple manipulation but once some helpers are created for that task they can be reused for all the operators.

The advantages are pretty obvious

1. Better typed operators and subscribers
2. Type safety for illegal operations (like take(4) on an Observable of size 2)

There are some disadvantages to this as well

1. This will probably slow down compile time due to added type complexity
2. Pushes the boundaries of the type system and new version of TypeScript may break RxJS due to this
3. Makes contributing to RxJS even more intimidating due to needing an expert level understanding of the type system.

Deciding to incorporate this doesn't mean RxJS needs to be rewritten and refactored. This solution can be applied one operator at a time, since the default Observable and operator will be still be size unknown pipelines. Also note that the existing usage of let a$: Observable<string | number> = of(1, 'test'); will still work since the hidden second type has a well defined default that will work as before when unspecified.

Choosing to implement this change should require buy in from the core TypeScript team since we'd need to push the limits of TS and want to ensure nothing we do is planned to be phased out.

As one of the largest TS libraries in the ecosystem and due to our usage of bleeding edge TS in order to make this library as intuitive as possible, a partnership with the TS team will benefit everyone involved.

/cc @benlesh @cartant @kwonoj

[kolodny]

I've started working on a repo to manipulate tuples at https://github.com/kolodny/tuplizer. We can use Take and Drop and Concat for the associated operators and can compose those and the other types for the most of the rest of the operators

[kolodny]

OK, I got this working well enough for to call this PoC working:

import { Take, LessThan } from 'tuplizer';

type FromArray<T extends any[]> = T extends Array<infer U> ? U : never;

class Observable<T, Ts extends T[] = T[]> {
  subscribe(callback: (t: T) => void) {}

  pipe(): Observable<T>;
  pipe<A, As extends A[]>(op1: OperatorFunction<T, A, Ts, As>): Observable<A, As>;
  pipe<A, As extends A[], B, Bs extends B[]>(op1: OperatorFunction<T, A, Ts, As>, op2: OperatorFunction<A, B, As, Bs>): Observable<B, Bs>;
  pipe(...operations: OperatorFunction<any, any>[]): Observable<any> {
    return {} as any;
  }
}

declare function of<T extends any[]>(...t: T): Observable<FromArray<T>, T>;

interface MonoTypeOperatorFunction<T, Ts extends T[] = T[]> extends OperatorFunction<T, T, Ts, Ts> {}
interface UnaryFunction<T, R> { (source: T): R; }
interface OperatorFunction<T, R, Ts extends T[] = T[], Rs extends R[] = R[]> extends UnaryFunction<Observable<T, Ts>, Observable<R, Rs>> {}

function take<N extends LessThan<Ts['length']>, T, Ts extends T[]>(
  count: N
): OperatorFunction<T, Take<Ts, N>[number], Ts, Take<Ts, N>> {
  return (source: Observable<T>) => {
    return {} as any;
  };
}

const foo = of(1, 'test', false, 2, 3);
const bar = foo.pipe(take(2));
declare function assert<T>(t: T): void;

// Got typings working well
assert<Observable<number|string, [number, string]>>(bar);

const baz = bar.pipe(take(4));
//                        ~ Argument of type '4' is not assignable to parameter of type '0 | 1'.

const baq = of(1 as const, '2' as const, '3', true).pipe(take(3), take(2));
assert<Observable<1|'2', [1, '2']>>(baq);

const apiCall$: Observable<number> = {} as any;
const tooken$ = apiCall$.pipe(take(1));
assert<Observable<number, [number]>>(tooken$);

[cartant]

I still haven't got around to writing down what I have in my head, but I did want to reference this issue/PR, as it's kinda related to this: #5072

[kolodny]

@cartant I tried using an intersection type instead of hidden tuple types as we discussed. I wasn't able to figure out how to easily extract the intersection type to facilitate knowing the elements and they're length. Below is the code I had before throwing in the towel. Perhaps you or some other TypeScript wizard can figure out how to get this working. Note in vscode you can collapse the initial magic types to focus on the actual Rx bits:

//#region Collapse me
export type SpecificNumber<N extends number> = number extends N ? never : N;
export type IsSpecificNumber<N extends number> = SpecificNumber<N> extends never ? false : true;
export type IsTuple<T extends any[]> = IsSpecificNumber<T['length']>;
type ExtractFromArray<T extends any[]> = T extends Array<infer U> ? U : never;
type Fn<Args extends any[]> = (...args: Args) => any;

export type Drop<T extends any[], N extends number> =
  IsTuple<T> extends false ? T :
  N extends  0 ? T :
  N extends  1 ? Fn<T> extends ((a: any, ...rest: infer R) => any) ? R : never :
  N extends  2 ? Fn<T> extends ((a: any, b: any, ...rest: infer R) => any) ? R : never :
  N extends  3 ? Fn<T> extends ((a: any, b: any, c: any, ...rest: infer R) => any) ? R : never :
  N extends  4 ? Fn<T> extends ((a: any, b: any, c: any, d: any, ...rest: infer R) => any) ? R : never :
  N extends  5 ? Fn<T> extends ((a: any, b: any, c: any, d: any, e: any, ...rest: infer R) => any) ? R : never :
  N extends  6 ? Fn<T> extends ((a: any, b: any, c: any, d: any, e: any, f: any, ...rest: infer R) => any) ? R : never :
  N extends  7 ? Fn<T> extends ((a: any, b: any, c: any, d: any, e: any, f: any, g: any, ...rest: infer R) => any) ? R : never :
  N extends  8 ? Fn<T> extends ((a: any, b: any, c: any, d: any, e: any, f: any, g: any, h: any, ...rest: infer R) => any) ? R : never :
  N extends  9 ? Fn<T> extends ((a: any, b: any, c: any, d: any, e: any, f: any, g: any, h: any, i: any, ...rest: infer R) => any) ? R : never :
  N extends 10 ? Fn<T> extends ((a: any, b: any, c: any, d: any, e: any, f: any, g: any, h: any, i: any, j: any, ...rest: infer R) => any) ? R : never :
  T[number][];

export type LessThan<N extends number> =
  IsSpecificNumber<N> extends false ? number :
  N extends  0 ? never :
  N extends  1 ? 0 :
  N extends  2 ? 0|1 :
  N extends  3 ? 0|1|2 :
  N extends  4 ? 0|1|2|3 :
  N extends  5 ? 0|1|2|3|4 :
  N extends  6 ? 0|1|2|3|4|5 :
  N extends  7 ? 0|1|2|3|4|5|6 :
  N extends  8 ? 0|1|2|3|4|5|6|7 :
  N extends  9 ? 0|1|2|3|4|5|6|7|8 :
  N extends 10 ? 0|1|2|3|4|5|6|7|8|9 :
  number;

export type Take<T extends any[], N extends number> =
  IsTuple<T> extends false ? MakeTuple<ExtractFromArray<T>, N> :
  Drop<T, N> extends [] ? T : // if taking more than T['length'] just return T;
  N extends  0 ? [] :
  N extends  1 ? [T[0]] :
  N extends  2 ? [T[0], T[1]] :
  N extends  3 ? [T[0], T[1], T[2]] :
  N extends  4 ? [T[0], T[1], T[2], T[3]] :
  N extends  5 ? [T[0], T[1], T[2], T[3], T[4]] :
  N extends  6 ? [T[0], T[1], T[2], T[3], T[4], T[5]] :
  N extends  7 ? [T[0], T[1], T[2], T[3], T[4], T[5], T[6]] :
  N extends  8 ? [T[0], T[1], T[2], T[3], T[4], T[5], T[6], T[7]] :
  N extends  9 ? [T[0], T[1], T[2], T[3], T[4], T[5], T[6], T[7], T[8]] :
  N extends 10 ? [T[0], T[1], T[2], T[3], T[4], T[5], T[6], T[7], T[8], T[9]] :
  T[number][];

export type MakeTuple<T, Size extends number> =
  Size extends  0 ? [] :
  Size extends  1 ? [T] :
  Size extends  2 ? [T, T] :
  Size extends  3 ? [T, T, T] :
  Size extends  4 ? [T, T, T, T] :
  Size extends  5 ? [T, T, T, T, T] :
  Size extends  6 ? [T, T, T, T, T, T] :
  Size extends  7 ? [T, T, T, T, T, T, T] :
  Size extends  8 ? [T, T, T, T, T, T, T, T] :
  Size extends  9 ? [T, T, T, T, T, T, T, T, T] :
  Size extends 10 ? [T, T, T, T, T, T, T, T, T, T] :
  T[]
//#endregion

declare const elements: unique symbol;

export type InternalElements<Elements extends unknown[] = unknown[]> = {
  [elements]: Elements;
};
export type ExtractElements<T> = T extends { [elements]: infer U } ? U : unknown[];

class Observable<T> {
  subscribe(callback: (t: T) => void) {}

  pipe(): this;
  pipe<A, TElements extends InternalElements, AElements extends InternalElements>(op1: OperatorFunction<T, A, TElements, AElements>): Observable<A> & AElements;
  pipe<A, B, TElements extends InternalElements, AElements extends InternalElements, BElements extends InternalElements>(op1: OperatorFunction<T, A, TElements, AElements>, op2: OperatorFunction<A, B, AElements, BElements>): Observable<B> & BElements;
  pipe(...operations: OperatorFunction<any, any>[]): Observable<any> {
    return {} as any;
  }
}

declare function of<T extends any[]>(...t: T): Observable<ExtractFromArray<T>> & InternalElements<T>;

interface UnaryFunction<T, R> { (source: T): R; }
interface OperatorFunction<T, R, TElements extends InternalElements = InternalElements, RElements extends InternalElements = InternalElements> extends UnaryFunction<Observable<T> & TElements, Observable<R> & RElements> {}

function take<T, TElements extends InternalElements, N extends LessThan< ExtractElements<InternalElements>['length'] >>(
  count: N
): OperatorFunction<T, T, InternalElements, TElements> {
  return (source: Observable<T>) => {
    return {} as any;
  };
}

const foo = of(1, 'test', false, 2, 3);
const thisFoo = foo.pipe();
thisFoo // returning this type works
const bar = foo.pipe(take(7)); // hmm this should error with something like `Argument of type '7' is not assignable to parameter of type '0 | 1 | 2 | 3 | 4'.`
declare function assert<T>(t: T): void;
foo
bar

[MaximSagan]

A shame to see the discussion end and the "agenda item" tag removed. This would be great.

@kolodny, would the implementation be simplified with the introduction of variadic tuple types (added in Typescript 4.0, after the last posts in this discussion)?

[kolodny]

This work is moving forward in https://github.com/cartant/eslint-plugin-rxjs-traits