Upgrade azure-eventhub to input v2 API

[zmoog]

Proposed commit message

Upgrade azure-eventhub to input v2 API.

The input v2 API provides better health status reporting and control. This is a preliminary step in introducing support for the modern Azure Event Hub SDK for Go.

Notes for reviewers

This PR aims to update the azure-eventhub input to the input v2 API without introducing behavior change. The PR is a stepping stone to #33815 (the PR is coming)

Here is a list of changes to keep in mind during the review.

The PR:

• Renames the azureInput to eventHubInputV1 because I want to continue offering the legacy SDK while we introduce an eventHubInputV2 with the modern event hub SDK in the next PR. The eventHubInputManager will select the input version depending on the configuration.
• Switches the pipeline publishing process from Outlet (input v1) to PipelineClient (input v2).
• Starts the EPH and stops it when the input context is "done".
• Rename parseMultipleMessages() to unpackRecords() because seems to better represent the behavior.

Keep existing behavior: Beats pipeline acks

It's important to note that the original input version did not handle publishing ACKs to the Beats pipeline. This was primarily because (AFAIK) the input API v1 did not provide this functionality.

With the transition to the input API v2, we have acks management. However, the legacy event hub SDK internally updates the checkpoint info after a successful handler call and does not seem to offer hooks to plug proper ACK management.

Since the new modern SDK offers better checkpoint management, in this PR, we keep the current behavior intact, and we'll implement ACKs in the event hub input v2.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• Test reload configuration

How to test this PR locally

Related issues

• Closes [Azure] Upgrade azure-eventhub to Input API v2 #36377
• Relates [Azure] Migrate the azure-eventhub input to the new Azure Event Hubs Client Module for Go #33815

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @zmoog? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[belimawr]

I took a quick look at the InputV2 API part of the PR and it looks good. I just have a question about an empty file.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b zmoog/azure-eventhub-to-input-api-v2 upstream/zmoog/azure-eventhub-to-input-api-v2
git merge upstream/main
git push upstream zmoog/azure-eventhub-to-input-api-v2