Keep the AWS/Azure/GCP SDK version up to date

[zmoog]

Situation

All major CSPs provide SDKs for accessing their API. Beats use the cloud provider API to collect logs, metrics, and metadata.

Over time, CSPs release new versions of SDK. Most of the time, new versions are minor or patch releases. Major releases with breaking changes usually happen every few years.

Usually, we upgrade the cloud provider SDK in two circumstances:

• we need a new feature
• we need a bugfix

That's only available in a new SDK version.

Our attitude is primarily reactive.

Problem statement

The current reactive posture has a few downsides:

• On average, our SDK modules are outdated to various degrees (missing fixes)
• Upgrades happen not so often, sometimes taking big jumps in versions (increasing risks)
• On average, if we need a bugfix in one of our dependencies, we can add it now, but the subsequent stack releases may be weeks away; we only backport to the previous release.

Solutions

Manage AWS/Azure/GCP SDK version incrementally using Dependabot.

Pros:

• It is more manageable to upgrade 1-2 dependencies at a time instead of doing a big batch occasionally.
• SDKs are up to date with fixes and improvements
• We integrate updates in the next release to improve the change or avoid bug reports and support requests.

Cons:

• We are making more changes; we need to mitigate this risk by improving our test suite (we'll address this in a dedicated issue).

Tasks

Beta Give feedback

1. Manage AWS SDK dependencies with Dependabot
2. Manage Azure SDKs version with Dependabot #39495
   Team:obs-ds-hosted-services +1
   
3. Manage GCP SDK dependencies with Dependabot

Related

Here are a few related issues and PRs:

• Fix the AWS SDK dependencies issue causing the "not found, ResolveEndpointV2" error #39454

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[zmoog]

Here is how APM Server manages OTel dependencies:
https://github.com/elastic/apm-server/blob/main/.github/dependabot.yml

[zmoog]

I'm creating the first PR to manage the Azure SDK to get feedback from the team responsible for Dependabot.

[agithomas]

If we notice the issue mentioned here, a sequential upgrade from working -> broken SDK (Nov '23 -> Feb'24) , it would have taken the application into a less desirable state.

So, the scope may be extended or changed to qualifying the SDK version. Addressing dependencies, Testing & verification, addressing gaps in testing (if any) may be part of the qualification process. Once qualified, how soon should we consume the upgrade?

[zmoog]

If we notice the issue mentioned here, a sequential upgrade from working -> broken SDK (Nov '23 -> Feb'24) , it would have taken the application into a less desirable state.

So, the scope may be extended or changed to qualifying the SDK version. Addressing dependencies, Testing & verification, addressing gaps in testing (if any) may be part of the qualification process. Once qualified, how soon should we consume the upgrade?

@agithomas, what do you mean by "qualifying the SDK version"? Can you elaborate a little on this?

[zmoog]

To be clear, the goal of PRs like #39495 is not to avoid problems like "not found, ResolveEndpointV2" but to ensure Beats remains up to date with new features, possibly avoid support requests, or at least ship the fixes sooner.

I agree that to avoid problems like "not found, ResolveEndpointV2" from happening we need a holistic strategy that includes:

• improved dependency management
• improved testing

[agithomas]

@agithomas, what do you mean by "qualifying the SDK version"? Can you elaborate a little on this?

My comment was more towards the holistic strategy.

Thanks for adding the scope here.