Consider splitting up the package

[olafurnielsen]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

• From the premise of the Python SDK with VScode but similar problems probably apply to other language SDKs and IDEs
• pulumi_azure_native contains >200 top-level namespaces, around 20k Python files and huge amount of symbols
• In comparison our fairly extensive IAC project focused on application/networking/security workloads is maybe using 20-30 of those top-level namespaces
• IDEs index those packages to aid automatic imports, code completion, input validation etc.
• If index depth in VScode is configured to index the whole library the Python language server will run out of memory (consumption above the 4GB hard-limit) and blow up
• Limiting index depth to only 2 levels (thus skipping any versioned API resources) won't make the Python language server crash but CPU load is constant 25% with spikes up to 90% when the indexer is triggered (opening up a Pulumi project, importing new packages)

Another option could be to offer both the all included package and then the namespace specific packages?

For reference the Azure SDK does this for most if not all of its supported languages, that is splitting the package based on provider namespaces.

Affected area/feature

• pulumi_azure_native
• Library build, packaging and publishing

[mjeffryes]

You're absolutely right @olafurnielsen (and indeed we do have some related issues on other SDKs #1981). We have a bit of groundwork to do to enable this (namely pulumi/pulumi#15207), but rest assured that we are aware of the problems with the size of the SDK and have this in our roadmap