Skip to content

Latest commit

 

History

History
218 lines (159 loc) · 12.9 KB

L70-node-proto-loader-type-generator.md

File metadata and controls

218 lines (159 loc) · 12.9 KB
Raw

@grpc/proto-loader TypeScript Type Generator CLI Tool

Abstract

Add a tool to the @grpc/proto-loader to generate types that describe the objects that will be generated by loading the output of @grpc/proto-loader into grpc or @grpc/grpc-js.

Background

@grpc/proto-loader outputs objects with types determined by the .proto files loaded at runtime, so it is currently very difficult to get compile-time type information about those objects. As TypeScript becomes increasingly popular, that compile-time type information becomes increasingly desirable.

Related Proposals:

  • L23 Standalone Node.js gRPC+Protobuf.js Library API
  • L43 Node Message Type Information

Proposal

In the @grpc/proto-loader library, provide a command line tool that will generate the types of the objects that will be created by passing the output of load or loadSync to grpc.loadPackageDefinition when loading a specific set of .proto files with a specific set of options. This will allow users to write code like this to use the type information in the rest of the code:

import * as grpc from 'grpc';
// OR 
import * as grpc from '@grpc/grpc-js';

import { ProtoGrpcType } from 'generated/proto-file-name_proto'; // File generated by the tool is generated/proto-file-name_proto.ts
import * as protoLoader from '@grpc/proto-loader';

import { MessageName } from 'generated/fully/qualified/package/MessageName';
import { ServiceNameHandler } from 'generated/fully/qualified/package/ServiceName';

const packageDefinition = protoLoader.loadSync('other/path/to/proto-file-name.proto', options);
const loadedPackageDefinition = grpc.loadPackageDefinition(packageDefinition) as unknown as ProtoGrpcType; // How the types will be used

const requestObject: MessageName = {
  field1: value1,
  field2: value2
};

const serviceHandler: ServiceNameHandler = {
  // Call implicitly has the correct arity (unary, client streaming, etc.) and the call and callback have the correct request and response types
  methodName(call, callback) {
    // Implementation
  }
}

const server = new grpc.Server();
server.addService(loadedPackageDefinition.fully.qualified.package.ServiceName.service, serviceHandler);

The tool will be named proto-loader-gen-types and will have this usage:

proto-loader-gen-types [OPTIONS] <proto file name> ...

The options will correspond directly to the protoLoader.load options as follows:

  • --keepCase: Preserve the case of field names

  • --longs=String|Number: Specify the type that should be used to output 64 bit integer values

  • --enums=String: Specify that enum values should be output as strings

  • --bytes=Array|String: Specify the type that should be used to output bytes fields

  • --defaults: Indicates that default values should be output for missing fields

  • --arrays: Indicates that empty arrays should be output for missing repeated fields even if --defaults is unset

  • --objects: Indicates that empty objects should be output for missing message fields even if --defaults is unset

  • --oneofs: Indicates that virtual "oneof" fields will be set to the present field's name in the output

  • --includeDirs=<directory>, -I <directory>: A directory to search for included .proto files. Can be passed multiple times to include multiple directories

  • --outDir=<directory>, -O <directory>: The directory in which to output files

  • --grpcLib=grpc|@grpc/grpc-js: The gRPC implementation library that these types will be used with

  • --verbose, -v: Enable various logging output

  • --includeComments: Include comments from the .proto files in the generated files

The output will be one file for each message and enum loaded, with file paths based on the package and type names, plus a master file per input file that combines all of those to produce the type that the user will load, as described above. ProtoGrpcType types from different files can be intersected to get the type that results from loading those files together at runtime. Messages will have an additional type generated, suffixed with __Output, that describes the type of objects that will be output by gRPC, i.e. response messages on the client, and request messages on the server. These "output" message types will be subtypes of the main message type, restricted based on the options that are set.

To support this generated code, @grpc/proto-loader will need to re-export the Long type from protobufjs, because that is a type that can be used by generated message types.

Example Generated Code

With options that will probably be common: --keep-case, --longs=String, --enums=String, --defaults, --oneofs, and --grpcLib=@grpc/grpc-js

Input

// filename.proto
syntax = "proto3";

package package_name.subpackage_name;

enum EnumName {
  OPTION0 = 0;
  OPTION1 = 1;
}

message MessageName {
  string string_value = 1;
  int32 number_value = 2;
  EnumName enum_value = 3;
  int64 long_value = 4;
  oneof oneof_value {
    bool bool_value = 5;
    bytes bytes_value = 6;
  }
}

service ServiceName {
  rpc Method (MessageName) returns (MessageName);
}

Output

// package_name/subpackage_name/EnumName.ts

export enum EnumName {
  OPTION0 = 0,
  OPTION1 = 1,
}

// package_name/subpackage_name/MessageName.ts

import { EnumName as _package_name_subpackage_name_EnumName } from '../../package_name/subpackage_name/EnumName';
import { Long } from '@grpc/proto-loader';

export interface MessageName {
  'string_value'?: (string);
  'number_value'?: (number);
  'enum_value'?: (_package_name_subpackage_name_EnumName | keyof typeof _package_name_subpackage_name_EnumName);
  'long_value'?: (number | string | Long);
  'bool_value'?: (boolean);
  'bytes_value'?: (Buffer | Uint8Array | string);
  'oneof_value'?: "bool_value"|"bytes_value";
}

export interface MessageName__Output {
  'string_value': (string);
  'number_value': (number);
  'enum_value': (keyof typeof _package_name_subpackage_name_EnumName);
  'long_value': (string);
  'bool_value'?: (boolean);
  'bytes_value'?: (Buffer);
  'oneof_value': "bool_value"|"bytes_value";
}

// package_name/subpackage_name/ServiceName.ts

import * as grpc from '@grpc/grpc-js'
import { MessageName as _package_name_subpackage_name_MessageName, MessageName__Output as _package_name_subpackage_name_MessageName__Output } from '../../package_name/subpackage_name/MessageName';

export interface ServiceNameClient extends grpc.Client {
  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
  
}

export interface ServiceNameHandlers {
  Method(call: grpc.ServerUnaryCall<_package_name_subpackage_name_MessageName, _package_name_subpackage_name_MessageName__Output>, callback: grpc.sendUnaryData<_package_name_subpackage_name_MessageName__Output>): void;
  
}

// filename.ts

import * as grpc from '@grpc/grpc-js';
import { ServiceDefinition, EnumTypeDefinition, MessageTypeDefinition } from '@grpc/proto-loader';

import { ServiceNameClient as _package_name_subpackage_name_ServiceNameClient } from './package_name/subpackage_name/ServiceName';

type ConstructorArguments<Constructor> = Constructor extends new (...args: infer Args) => any ? Args: never;
type SubtypeConstructor<Constructor, Subtype> = {
  new(...args: ConstructorArguments<Constructor>): Subtype;
}

export interface ProtoGrpcType {
  package_name: {
    subpackage_name: {
      EnumName: EnumTypeDefinition
      MessageName: MessageTypeDefinition
      ServiceName: SubtypeConstructor<typeof grpc.Client, _package_name_subpackage_name_ServiceNameClient> & { service: ServiceDefinition }
    }
  }
}

Rationale

The goal is to have type information available when editing and building code that uses @grpc/proto-loader to load .proto files at runtime. It is not possible to infer this type information from the interfaces currently provided by this library because TypeScript cannot infer type information from .proto files. The only other option is to generate this type information separately. The recommended usage of the @grpc/proto-loader library is to call load or loadSync and then pass the result of that to grpc.loadPackageDefinition, so the type of the final result of that will be useful to most users. The output of load and loadSync are objects with runtime data that describes the resulting types, so the types of those objects are significantly less useful, and the final type that we will generate here cannot be effectively inferred from those types.

The type parameters grpc.Client, grpc.Metadata, and grpc.CallOptions are all needed because each of those types is used in the resulting type, and none of them can be inferred from the others. The type grpc.Metadata also impacts the resulting type, but its type is the same across the two implementations so it can be defined independently of them.

These alternatives were considered to generating files in a directory structure based on the protobuf package structure:

  • Generate files in a directory structure mirroring the directory structure of the input files, all relative to the output directory. This introduces the complexity of handling unusual import paths, including absolute paths and paths with ...
  • Generate files corresponding to the input files, all directly in the output directory. This greatly increases the risk of filename conflicts.
  • Generate a single file with all of the types. This creates a large, unwieldy file, making it hard for a human to evaluate changes. This can also greatly increase code duplication if multiple files need to be generated.

The gRPC implementation is specified at build time because the types depend on types from that library and the user should know at that point which implementation they are using. An alternative is to use generics to insert the types in a different way, but that increases the complexity of both the generated code and the code that uses it for relatively little gain.

The goal of generating two separate interfaces for each message type is to describe two separate things as narrowly as possible: the objects that users can pass to the library as input, and the objects that the library will output. We have more control over what the library outputs, so we can be more specific in that type. This simplifies handling of messages output by the library, while still allowing the same flexibility when providing input messages that users get with the JavaScript interface.

For example, all Protobuf 3 fields are optional, so the input type allows the user to omit fields, but with the defaults code generation option, we know that the library will always output the default value for omitted fields, so the output type can guarantee that every field will have a value.

Implementation

I (murgatroid99) will implement this in the @grpc/proto-loader library in PR #1474