[CT-896] [Feature] Add Jinja function which returns all `DBT_ENV_CUSTOM_ENV_ ` and `DBT_CLOUD_` environment variables

[NiallRees]

Is this your first time opening an issue?

• I have read the expectations for open source contributors

Describe the Feature

Right now, you have to know which environment variables might be set in order to get the values for them using the env_var function. For the dbt_artifacts package (and other like it) it would be helpful to be able to call a single function which returns all DBT_ENV_CUSTOM_ENV_ and DBT_CLOUD_ environment variables, in the same way they are all present in the manifest.json's metadata env field.

Describe alternatives you've considered

Asking users to configure variables to describe their in-use env vars

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

[jtcohen6]

Legit! Thanks for opening @NiallRees.

I don't think there would be any security concerns related to this, as long as we keep it tightly locked down to just the environment variables with the appropriate prefix.

It just needs to be DBT_ENV_CUSTOM_ENV_: That's all we document here, and have the logic for here:

dbt-core/core/dbt/contracts/util.py

Lines 153 to 161 in 94a7cfa

	METADATA_ENV_PREFIX = "DBT_ENV_CUSTOM_ENV_"
	def get_metadata_env() -> Dict[str, str]:
	return {
	k[len(METADATA_ENV_PREFIX) :]: v
	for k, v in os.environ.items()
	if k.startswith(METADATA_ENV_PREFIX)
	}

The way that dbt Cloud sets the environment variables is: DBT_ENV_CUSTOM_ENV_DBT_CLOUD_JOB_ID.

We have a few different definitions of the env_var context method, in order to ensure that it works properly with partial parsing. (We need to know whether the env_var call happened in a yaml file, a SQL file, or the profile / project file, in order to know what needs re-parsing when the env var value changes.) I think we do not care about partial parsing for this method: The whole point is just to pass along metadata, not to use those metadata vars to actually set project configuration.

So, I'd be inclined to say:

• We add metadata_env() as a function in the ProviderContext, a.k.a. the runtime context. It won't be accessible at parse time, which is intentional. (I could be convinced by a member of Core-Language that the BaseContext is just better for simplicity, though.)
• It returns exactly the value of get_metadata_env() above. Heck, it may as well import and use the function.

I'm going to mark this a help_wanted, since it will also require updating a few tests. Is that something you'd be interested in contributing?