Typescript Crashing

[XavierGeerinck]

What happened?

Typescript keeps on crashing and is stuck on Initializing JS/TS language features a bug was opened on the Typescript repo as well already, but I believe that it might be interesting to add Pulumi to it as Pulumi is the cause due the huge number of dependencies that are being loaded.

Currently, Typescript is NOT usable until this gets resolved as type suggestion is not working.

More info: microsoft/TypeScript#50809

Steps to reproduce

This demonstrates the issue (after a while, while working with it and trying to get command completion, reproduced on Mac and Windows through WSL)

import * as pulumi from "@pulumi/pulumi/index.js";
import * as resources from "@pulumi/azure-native/resources/index.js";

export const glbLocation = config.get("glb-location");
export const glbProjectName = config.get("glb-project-name");
export const glbProjectEnv = config.get("glb-project-env"); // prd, stg, dev, tst, tmp, ...

// ======================================================================
// Create the main Resource Group
// ======================================================================
const resourceGroup = pulumi.all([glbLocation, glbProjectName, glbProjectEnv]).apply(([glbLocation, glbProjectName, glbProjectEnv]) => {
  return new resources.ResourceGroup(`rg-${glbProjectName}-${glbProjectEnv}-`, {
    location: glbLocation
  });
})

Expected Behavior

Working type suggestions

Actual Behavior

Crashing TS language features

Output of pulumi about

CLI          
Version      3.40.0
Go Version   go1.19
Go Compiler  gc

Host     
OS       ubuntu
Version  20.04
Arch     x86_64

Additional context

Tested on Linux and Mac and neither are working

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[lblackstone]

pulumi/pulumi#10442 may help with this.

@XavierGeerinck Is pulumi up failing here, or is this specifically for type information in your editor?

[XavierGeerinck]

This is specifically for type information in the editor

[XavierGeerinck]

Any information on this? Currently still unable to utilize Pulumi on any of my PCs (Windows, Linux or Mac) since it keeps on initializing Typescript

[Ankvi]

Having the same issue. IntelliSense never finishes loading. Removing the @pulumi/azure-native package immediately fixed the problem. Initializing JS/TS features finished within a few seconds instead of never finishing at all.

[XavierGeerinck]

I think this will be solved through pulumi/pulumi#10831? so let's see after it is merged

[itays-chase]

Any update? can't develop using this module

[XavierGeerinck]

So far, no update, I switched to Mac for now as there it is working at least a bit, but on Windows (using WSL - thus Linux) TS keeps crashing. Checked with the TS team and they state clearly that it's not their responsibility as Pulumi loads too many dependencies.

[itays-chase]

Yeah, it does seems like a Pulumi issue.
I wonder what can be done, because for me, it's impossible to develop like this. @danielrbradley

[praneetloke]

While not optimal, I've been using a different IDE as a workaround for my project. I realize it may not be an option for others to switch to a different IDE, but if you can, JetBrains' Fleet (free while in Preview) or WebStorm (paid product) work well when using azure-native. It's not ideal but at least I can use azure-native without the TS LSP crashing often.

[danielrbradley]

I think this issue is already captured by #913 and #932 so might close as a duplicate to keep the conversation in a single thread.

This is a fairly complex issue to solve at the code generation level and requires being done in a backward compatible manner. We've merged some groundwork in this area recently but there's still additional follow-up work in the core code-gen which we'll then need to incorporate.

Apologies there's not more we can do more immediately.

[Ankvi]

As this is an issue caused by a giant npm module, it can be improved by simply removing stuff that's not needed without removing anything that's required by other files. For my team, I created a script that simply locates all local folders in node_modules named e.g. v20201005 and replaced it with a index.js file with a default module export.

We don't use anything in the older versions and they don't seem to be directly imported by other modules so we haven't had any issues so far. I realize that this is a sketchy solution to a problem that can just cause a bunch of other issues if you use the Pulumi CLI more than we do, but it works for us.

I'm considering expanding this script further to remove modules that we don't use in the same way. For instance, if we only use apimanagement, documentdb and servicebus, I will try to simply clear out all other folders with an empty default export and see if that works. Ideally, I shouldn't have to do this, but by removing the folders containing older versions, the TS server of my laptop stopped crashing.

If you're not completely horrified by this temporary fix, I can share the script I wrote

[danielrbradley]

@Ankvi love the creative approach! 😬

While officially we wouldn't support or maybe even recommend doing this, I have no issue sharing workarounds untill we have a more perminant solution

[antoine-lecomte]

Hello @Ankvi I'm exploring temporary solutions between JetBrain Fleet and yours.

Can you kindly provide an example of your script ?
Thanks 😄

[praneetloke]

I don't know if VS Code has made some improvements recently but it has been working quite well for me. I have one project that's using 1.78.0 and another 1.88.0 versions of the SDK, both seem to be working fine. I am on VS Code version 1.74.0 that was released recently. I've been using it on both Win 11 and macOS (M1). In previous versions, I could barely open a file or two before the LSP crashed and I would have to reload the editor. I'll update this comment as I continue to use VS Code over the next few days on both platforms.

[martinmine]

Having looked a bit into this, there is a high chance other languages than JS/TS will experience similar problems. As @Ankvi briefly mentions, each module has separate generated directories for different API versions. So for each new version that is pushed out, it will inherit everything all the previous versions as well, which is the root-cause for this issue. For example, the Authorization module has over 30 versions bundled together with it (I'm sure there are bigger ones): https://github.com/pulumi/pulumi-azure-native/tree/master/sdk/nodejs/authorization
I do not know why these packages are put together this way. Is there a dependency between each version? I'm really curious to know since this approach to versioning is obviously going to cause problems for people.

[Ankvi]

Hi! Sorry, completely forgot about this over the holidays etc etc.
Here's my current script. It's not the prettiest piece of code I've written in my career, but hopefully I'll be able to explain.

So in short, this script will loop over all folders in the @pulumi/azure-native package and subsequently delete all folders not explicitly marked in the modulesToKeep variable. This is of course possible to make more dynamic but as this just lurks in our repo, it'll stay the same until we decide to include another package.

If the module is going to be kept, it will find all version folders inside that module and perform the same replacement script in there.

The exported function is used by an index file in our case which is executed via command line, but you're free to do whatever suits your use case best. I decided to write it in TypeScript to be in the same format as all our other non .NET code, but it should be fairly simple to turn this into regular JavaScript as the only types present are a few input variables.

The final exported function is a desperate attempt to make things go a little bit quicker by utilizing promises for each folder and execute them in parallel.

Hopefully this will help you all out!

import { readdir, rm, writeFile } from "fs/promises";

const rootFolder = `${__dirname}/../../node_modules/@pulumi/azure-native`;
const enumsFolder = `${rootFolder}/types/enums`;

const modulesToKeep = [
    "documentdb",
    "types",
    "apimanagement",
    "servicebus",
    "keyvault",
    "config"
];

async function getPreviousVersionFolders(path: string) {
    const moduleFolders = await getFolderNames(path);
    return moduleFolders
        .filter(x => {
            const match = x.match(/v\d{8}(?:preview)?$/);
            return match && match.length === 1;
        })
        .map(x => [path, x].join("/"));
}

async function getFolderNames(path: string) {
    const dirents = await readdir(path, {
        withFileTypes: true
    });

    return dirents
        .filter(x => x.isDirectory())
        .map(x => x.name);
}

async function getFolders(rootPath: string) {
    const modules = await getFolderNames(rootPath);

    const foldersPromises = modules.map(async (module) => {
        const moduleFolderPath = [rootPath, module].join("/");
        if (modulesToKeep.includes(module)) {
            return await getPreviousVersionFolders(moduleFolderPath);
        }
        return moduleFolderPath;
    });

    const moduleFolders = await Promise.all(foldersPromises);
    return moduleFolders.flat();
}

async function getFoldersToClean() {
    const getFoldersPromises = [rootFolder, enumsFolder].map(getFolders);
    const folders = await Promise.all(getFoldersPromises);
    return folders.flat();
}

export async function cleanFoldersAsync() {
    const folders = await getFoldersToClean();

    const removeTasks = await Promise.all(folders.map(async folder => {
        const files = await readdir(folder, {
            withFileTypes: true
        });

        return files.map(file => {
            const path = `${folder}/${file.name}`;
            return rm(path, {
                recursive: true,
                force: true
            });
        });
    }));

    const removeTasksFlattened = removeTasks.flat();
    await Promise.all(removeTasksFlattened);

    await Promise.all(folders.map(async folder => {
        const newIndexFile = [folder, "index.js"].join("/");
        await writeFile(newIndexFile, 'Object.defineProperty(exports, "__esModule", { value: true });');
    }));
}

[joeskeen]

Also logged an issue with VSCode but it turned out to be because of @pulumi/azure-native

I tried Ubuntu, Windows, VSCode, VSCode insiders... crashed my project no matter what I did. IMHO this package is unusable until the issue is resolved. Not sure why it doesn't seem to be a priority.

see microsoft/vscode#173434

[joeskeen]

I created a version of @Ankvi's script for my purposes, cleaning up a couple things and making it automatically run after npm install. To use it, follow these instructions:

• add the following to your "scripts" (or create the "scripts" entry if it does not exist) in package.json:

  {
      "scripts": {
          "postinstall": "npx ts-node scripts/postinstall-cleanup.ts"
      }
  }

• paste the code from the following Gist into scripts/postinstall-cleanup.ts: https://gist.github.com/joeskeen/fe3081a088653140171e4bc55a579134

[antoine-lecomte]

Hello,

just wanted to share an improved version of @Ankvi adapted to 2 of our needs :

• Keep the default and specifics versions of a resource (ex : v20210301)
• Allow to clean in packages that include @pulumi/azure-native

The packageModulesToKeep have the following format :

/**
 * You can indicate multiples packages
 * Specify modules to keep (and optional version, the default version will be kept)
 */
const packageModulesToKeep: PackageToKeep = {
    '': {
        // Main project
        config: [],
        types: [],
        resources: [],
        web: ['v20210301'],
    },
    'node_modules/submodulename': {
        // Sub-module have its own @pulumi/azure-native
        config: [],
        types: [],
        authorization: ['v20200801preview'],
    }
}

The full script can be seen here : https://gist.github.com/antoine-lecomte/db85d01ad2e57d5e786a68f8d5294a99

[brucetan-displayr]

setting typescript.tsserver.maxTsServerMemory to 4096 in VS code seems to solve this problem for me

[iBezosS]

Hello,
Any news regarding this issue? I'm experiencing it and it slows pulumi up substantially since adding azure-native and the vscode project itself is barely usable.
Would happily provide debug logs.

[danielrbradley]

Version 2.0.0 of the provider has now been released. This significantly reduces the size of the Node.js SDK which should resolve these issues with the TSServer crashing.

Thank you for your patience as we've been preparing this major release. I'm going to close this issue out as resolved now but please do get in touch if you encounter any issues with the 2.0 provider.