Add Apache Pulsar Support to testcontainers-dotnet with Option for TokenAuthentication

[entvex]

What does this PR do?

This PR introduces Apache Pulsar support to testcontainers-dotnet, enabling users to easily spin up Pulsar containers for testing purposes. It also provides an option to enable TokenAuthentication, allowing for secure Pulsar interactions.

Why is it important?

Apache Pulsar is a rapidly growing distributed messaging system gaining traction in the cloud-native ecosystem. Integrating Pulsar support into testcontainers-dotnet will make it easier for developers to test their applications using Pulsar. Additionally, providing TokenAuthentication support enhances security and privacy in Pulsar-based testing scenarios.

How to test this PR

Run the XUnit tests in the testcontainers-dotnet project to ensure the Pulsar containerization and TokenAuthentication functionality work as expected.

Follow-ups

@HofmeisterAn, I encountered a challenge while adding WithTokenAuthentication capabilities to PulsarBuilder. The current implementation of WithCommand appends commands instead of replacing them, making it difficult to modify container startup commands based on selected Pulsar features. In the testcontainers-java module, the container command is modified just before the container starts, enabling control over its execution. Could you provide suggestions on how to achieve a similar approach in testcontainers-dotnet?

In the java module they seem to have the control to edit it, just before the container starts

Do you have a suggestion on a suggestion on how I should implement this for testcontainers-dotnet ?

[netlify]

✅ Deploy Preview for testcontainers-dotnet ready!

Name	Link
🔨 Latest commit	fd96367
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-dotnet/deploys/6626677f986700000813d4e7
😎 Deploy Preview	https://deploy-preview-1103--testcontainers-dotnet.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[HofmeisterAn]

The current implementation of WithCommand appends commands instead of replacing them, making it difficult to modify container startup commands based on selected Pulsar features.

You can apply the final command at the end. We are doing this in other modules too; sometimes it is necessary to configure the wait strategy at the end. You just need to keep a reference somewhere (in the configuration) to decide which command should be used. Or you can use a similar approach we are doing with Kafka and upload a script that gets executed. Does that help?

[entvex]

The current implementation of WithCommand appends commands instead of replacing them, making it difficult to modify container startup commands based on selected Pulsar features.

You can apply the final command at the end. We are doing this in other modules too; sometimes it is necessary to configure the wait strategy at the end. You just need to keep a reference somewhere (in the configuration) to decide which command should be used. Or you can use a similar approach we are doing with Kafka and upload a script that gets executed. Does that help?

Yes! Thanks 👍

[entvex]

Hi @HofmeisterAn ,
Testcontainers.Pulsar is now ready for a code review.

[entvex]

Dear @HofmeisterAn.
I can see you have started looking at my PR.
Please let me know if you need any help getting it across the finish line 💪.

[entvex]

@HofmeisterAn,
I have successfully integrated the approach you suggested from Couchbase into the pulsar module. During my individual testing, the tests passed without any issues. However, when I queued them to run consecutively, I noticed that the test without authentication was hanging. I am wondering if there might be an issue with the [UsedImplicitly] after you refactored the tests. Could you please take a closer look and let me know if you spot anything?

Thank you for your valuable contribution to the open-source community.

[HofmeisterAn]

I am wondering if there might be an issue with the [UsedImplicitly] after you refactored the tests.

No, this controls the usage check (code smell detection).

The issue was with the EnabledServices implementation. You cannot store values in the builder because the builder returns new instances and loses the information without further adjustments (we made this change to support A/B testing). Therefore, the authentication use case was not properly configured. Please take a look at the latest adjustments and let me know what you think. IMO the implementation looks good. However, we should take a final look at the configuration and figure out why the latest version is incompatible. Version 3.2.2 does not start with enabled authentication.

[HofmeisterAn]

Support the latest version 3.2 too.

[entvex]

Thanks for catching the issue with Pulsar 3.2.2! We'll investigate this further, in the Pulsar community.

In the meantime, using Pulsar version 3.0.4 for TestContainers is a good choice. It's an LTS (Long-Term Support) version, which means it will be supported for a longer period compared to Pulsar 3.2.2 (feature release with shorter support ending in 3 months).

The builder's design for A/B testing makes sense. My local TestContainers tests have been running stably as well.

If you think it's ready. Feel free to merge 👍

[HofmeisterAn]

I would prefer to address the issue right at the beginning of a new module. Fixing it afterwards is usually difficult (without a module owner). While I had a quick look at the issue, it seems like there are two issues:

1. The /pulsar/secret.key file is not created when the wait strategy tries to create the auth token. This is simple to fix. Either wait for the file until it exists or catch the exception and handle the error.

2. When I wait until the key exists, and the token gets created successfully, I still receive occasionally: HTTP ERROR 401 Authentication required. Once in a while, it works. When the wait strategy does not cache the auth token, it seems less likely to fail. Of course, this is not a proper fix.

Both findings indicate some kind of race-condition, and I am not sure if this will happen with version 3.0.4 as well.

Edit: When I add a 5 second delay right at the beginning of the wait strategy (before creating the auth token), the tests seem to run reliable, which confirms the race-condition we need to fix first, I guess.

diff --git a/src/Testcontainers.Pulsar/PulsarBuilder.cs b/src/Testcontainers.Pulsar/PulsarBuilder.cs
index 7b9b835..c9aa35a 100644
--- a/src/Testcontainers.Pulsar/PulsarBuilder.cs
+++ b/src/Testcontainers.Pulsar/PulsarBuilder.cs
@@ -163,6 +163,9 @@ public sealed class PulsarBuilder : ContainerBuilder<PulsarBuilder, PulsarContai
             {
                 try
                 {
+                    await Task.Delay(TimeSpan.FromSeconds(5))
+                        .ConfigureAwait(false);
+
                     _authToken = await container.CreateAuthenticationTokenAsync(TimeSpan.FromHours(1))
                         .ConfigureAwait(false);
 
diff --git a/tests/Testcontainers.Pulsar.Tests/PulsarContainerTest.cs b/tests/Testcontainers.Pulsar.Tests/PulsarContainerTest.cs
index 27ae620..dd4def9 100644
--- a/tests/Testcontainers.Pulsar.Tests/PulsarContainerTest.cs
+++ b/tests/Testcontainers.Pulsar.Tests/PulsarContainerTest.cs
@@ -73,7 +73,7 @@ public abstract class PulsarContainerTest : IAsyncLifetime
     public sealed class PulsarAuthConfiguration : PulsarContainerTest
     {
         public PulsarAuthConfiguration()
-            : base(new PulsarBuilder().WithAuthentication().Build())
+            : base(new PulsarBuilder().WithAuthentication().WithImage("apachepulsar/pulsar:3.2.2").Build())
         {
         }
 

[Loydik]

Can you please complete this? This library is needed for my project. Thank you

[entvex]

Can you please complete this? This library is needed for my project. Thank you

I am waiting for a reponse from @HofmeisterAn :)

[HofmeisterAn]

I stumbled across this discussion where the friends from Go are trying to achieve something similar. Maybe this can help to sort out the issue or align the configuration.