Growing /tmp directory

[Anumrati]

What happened?

When trying to upgrade the operator from v1.13.0 to v1.15.0 we started having problems with the growing /tmp directory. In the earlier version of the operator, directories from /tmp/pulumi-working/pulumi-kubernetes-operator/ were deleted after reconciliation. After change in PR-490 the .pulumi directory, which weighs about 800 MB in our instance, remains in each stack. With about 200 stacks we have, we need over 100 GB of extra space, which is not the best option.

Example

Thu Mar 14 09:03:20 UTC 2024
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       124G   71G   53G  58% /tmp
Thu Mar 14 09:04:20 UTC 2024
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       124G   79G   46G  64% /tmp
Thu Mar 14 09:05:20 UTC 2024
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       124G   86G   39G  70% /tmp
Thu Mar 14 09:06:20 UTC 2024
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       124G   93G   32G  75% /tmp
Thu Mar 14 09:07:20 UTC 2024
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       124G  100G   24G  81% /tmp

Output of pulumi about

CLI
Version      3.109.0
Go Version   go1.22.0
Go Compiler  gc

Host
OS       debian
Version  11.9
Arch     x86_64

Backend
Name           pulumi.com
URL            https://app.pulumi.com
User           Unknown
Organizations
Token type     personal

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mjeffryes]

Thanks for filing this issue @Anumrati; it looks like we need a way to either separate the credentials configuration from the pulumi home dir (pulumi/pulumi#13919) as an alternate solution for #483 or support some sort of reuse/hardlinking of the pulumi binaries when using multiple pulumi home dirs.