Localize TypeScript Imports

[t0yv0]

Consider modifying TypeScript provider code generation to optimize for the number of definitions that TypeScript compiler needs to process during compilation of simple Pulumi programs that use resource providers.

Issue details

For a motivating example consider that ~13s is spent compiling TypeScript on a simple program that references an S3 Bucket. This is compared to ~2s spent on a program that does not reference any resources. Note how the compiler needs to process 1101447 lines of definitions vs 10 lines of TypeScript.

$ pulumi new aws-typescript
$ tsc --extendedDiagnostics    ~/tmp/my-perf-ts-aws-test/test2
Files:                         2354
Lines of Library:             26582
Lines of Definitions:       1101447
Lines of TypeScript:             10
Lines of JavaScript:              0
Lines of JSON:                    0
Lines of Other:                   0
Nodes of Library:            117113
Nodes of Definitions:       2834671
Nodes of TypeScript:             37
Nodes of JavaScript:              0
Nodes of JSON:                    0
Nodes of Other:                   0
Identifiers:                1070341
Symbols:                     649499
Types:                       283897
Instantiations:              285896
Memory used:               1126911K
Assignability cache size:     19810
Identity cache size:              4
Subtype cache size:               0
Strict subtype cache size:        0
I/O Read time:                0.36s
Parse time:                   4.52s
ResolveModule time:           0.28s
ResolveTypeReference time:    0.00s
Program time:                 5.39s
Bind time:                    1.73s
Check time:                   6.20s
transformTime time:           0.01s
Source Map time:              0.00s
commentTime time:             0.00s
I/O Write time:               0.00s
printTime time:               0.02s
Emit time:                    0.02s
Total time:                  13.35s

Current source code:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("my-bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

Optimizing the imports in this program gives:

import { Bucket } from "@pulumi/aws/s3";

// Create an AWS resource (S3 Bucket)
const bucket = new Bucket("my-bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

However, TypeScript compilation remains just as slow. Digging deeper, it appears that bucket.d.ts makes these references:

import { input as inputs, output as outputs, enums } from "../types";
import { PolicyDocument } from "../iam";

There is a way to debug loading of the .d.ts files tsc --traceResolution. It appears that the entire AWS set of definitions is imported. Judicious optimizations here can help ensure that a smaller set is imported; for example only S3 definitions.

digraph G {
"/Users/anton/tmp/my-perf-ts-aws-test/index.ts" -> "@pulumi/aws/s3/bucket.d.ts";
"@pulumi/aws/s3/bucket.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucket.d.ts" -> "@pulumi/aws/iam/index.d.ts";
"@pulumi/aws/types/index.d.ts" -> "@pulumi/aws/types/enums/index.d.ts";
"@pulumi/aws/types/index.d.ts" -> "@pulumi/aws/types/input.d.ts";
"@pulumi/aws/types/index.d.ts" -> "@pulumi/aws/types/output.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/alb/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/applicationloadbalancing/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/autoscaling/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/ec2/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/iam/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/lambda/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/rds/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/route53/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/s3/index.d.ts";
"@pulumi/aws/types/enums/index.d.ts" -> "@pulumi/aws/types/enums/ssm/index.d.ts";
"@pulumi/aws/types/input.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/types/input.d.ts" -> "@pulumi/aws/s3/index.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/accessPoint.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/accountPublicAccessBlock.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/analyticsConfiguration.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucket.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketAccelerateConfigurationV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketAclV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketCorsConfigurationV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketIntelligentTieringConfiguration.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketLifecycleConfigurationV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketLoggingV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketMetric.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketNotification.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketObject.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketObjectLockConfigurationV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketObjectv2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketOwnershipControls.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketPolicy.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketPublicAccessBlock.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketReplicationConfig.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketRequestPaymentConfigurationV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketServerSideEncryptionConfigurationV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketVersioningV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/bucketWebsiteConfigurationV2.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/cannedAcl.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getAccountPublicAccessBlock.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getBucket.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getBucketObject.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getBucketObjects.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getBucketPolicy.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getCanonicalUserId.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getObject.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/getObjects.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/inventory.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/objectCopy.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/routingRules.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/s3/s3Mixins.d.ts";
"@pulumi/aws/s3/index.d.ts" -> "@pulumi/aws/types/enums/s3/index.d.ts";
"@pulumi/aws/s3/accessPoint.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/analyticsConfiguration.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketAclV2.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketCorsConfigurationV2.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketIntelligentTieringConfiguration.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketLifecycleConfigurationV2.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketLoggingV2.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketMetric.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketNotification.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketObject.d.ts" -> "@pulumi/aws/s3/index.d.ts";
"@pulumi/aws/s3/bucketObjectLockConfigurationV2.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketObjectv2.d.ts" -> "@pulumi/aws/s3/index.d.ts";
"@pulumi/aws/s3/bucketOwnershipControls.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/s3/bucketPolicy.d.ts" -> "@pulumi/aws/iam/index.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/accessKey.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/accountAlias.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/accountPasswordPolicy.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/documents.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getAccountAlias.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getGroup.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getInstanceProfile.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getInstanceProfiles.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getOpenidConnectProvider.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getPolicy.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getPolicyDocument.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getRole.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getRoles.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getSamlProvider.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getServerCertificate.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getSessionContext.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getUser.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getUserSshKey.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/getUsers.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/group.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/groupMembership.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/groupPolicy.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/groupPolicyAttachment.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/instanceProfile.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/managedPolicies.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/openIdConnectProvider.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/policy.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/policyAttachment.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/principals.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/role.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/rolePolicy.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/rolePolicyAttachment.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/samlProvider.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/serverCertificate.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/serviceLinkedRole.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/serviceSpecificCredential.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/signingCertificate.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/sshKey.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/user.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/userGroupMembership.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/userLoginProfile.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/userPolicy.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/userPolicyAttachment.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/iam/virtualMfaDevice.d.ts";
"@pulumi/aws/iam/index.d.ts" -> "@pulumi/aws/types/enums/iam/index.d.ts";
"@pulumi/aws/iam/getGroup.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/iam/getPolicyDocument.d.ts" -> "@pulumi/aws/types/index.d.ts";
"@pulumi/aws/iam/groupPolicy.d.ts" -> "@pulumi/aws/iam/index.d.ts";
"@pulumi/aws/iam/groupPolicyAttachment.d.ts" -> "@pulumi/aws/index.d.ts";
"@pulumi/aws/iam/groupPolicyAttachment.d.ts" -> "@pulumi/aws/iam/index.d.ts";
"@pulumi/aws/index.d.ts" -> "@pulumi/aws/arn.d.ts";
"@pulumi/aws/index.d.ts" -> "@pulumi/aws/awsMixins.d.ts";
"@pulumi/aws/index.d.ts" -> "@pulumi/aws/getAmi.d.ts";
"@pulumi/aws/index.d.ts" -> "@pulumi/aws/getAmiIds.d.ts";
}

Affected area/feature

Node codegen.

[t0yv0]

Related: #8613

Note that working on this issue may also help break down large output.d.ts files by moving definitions out somewhere else and re-importing them, what 8613 implies. However, it is likely that to actually speed up TypeScript compilation we need to ensure that whether the files are small or large, there is a smaller number of definitions consulted during compilation. Essentially Nodes of Definitions on a few sample program like the templates is a better metric to optimize for.

[t0yv0]

@RobbieMcKinstry I've published an utility that might help here https://github.com/t0yv0/tsc-resolution-tracker