feat: add Qdrant module #1149

russcam · 2024-04-02T21:10:10Z

Closes #992

What does this PR do?

This commit adds a qdrant container to the list of supported Testcontainers, for use with the Qdrant vector database.

The qdrant container allows configuration of:

an API key to authenticate to Qdrant
a certificate and private key in PEM format used to secure communication to Qdrant with Transport Layer Security

Why is it important?

Provides a qdrant test container for folks that need it

Related issues

#992
Supersedes #994

How to test this PR

Run tests in Testcontainers.Qdrant.Tests

This commit adds a qdrant container to the list of supported Testcontainers. The qdrant container allows configuration of: - an API key to authenticate to Qdrant - an x509 certificate used to secure communication to Qdrant with Transport Layer Security - a custom configuration file. See https://qdrant.tech/documentation/guides/configuration/ Closes testcontainers#992

Qdrant uses rustls for TLS, which does not accept IPv4 or IPv6 addresses as the hostname in SNI, adhering to RFC 6066: https://www.rfc-editor.org/rfc/rfc6066#page-7 Sending an IPv4 or IPv6 address as the hostname in SNI causes the handshake to be rejected and qdrant to log WARN rustls::msgs::handshake: Illegal SNI hostname received "127.0.0.1" By default, .NET on linux sends the DNS name in the URI in Server Name Indication (SNI), irrespective of whether it's an IP address or not. In order to resolve this, a custom Host request header is added, per dotnet/runtime#20876 (comment)

netlify · 2024-04-02T21:10:33Z

✅ Deploy Preview for testcontainers-dotnet ready!

Name	Link
🔨 Latest commit	`f31250e`
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-dotnet/deploys/6613d2a9f9b1cb000875ac68
😎 Deploy Preview	https://deploy-preview-1149--testcontainers-dotnet.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

eerhardt · 2024-04-04T17:48:56Z

src/Testcontainers.Qdrant/Testcontainers.Qdrant.csproj

@@ -0,0 +1,12 @@
+<Project Sdk="Microsoft.NET.Sdk">
+    <PropertyGroup>
+        <TargetFrameworks>net6.0;net8.0;netstandard2.0;netstandard2.1</TargetFrameworks>


Are all these really necessary? I would think the following would be enough:

Suggested change

<TargetFrameworks>net6.0;net8.0;netstandard2.0;netstandard2.1</TargetFrameworks>

<TargetFrameworks>net6.0;netstandard2.0</TargetFrameworks>

Every other TestContainer module defines these target frameworks, so was sticking to the project convention. I've also added net462 for same reason Milvus module does: #1131 (comment)

I think we should stick to the repo convention too 👍. In the end, it does not really matter, right (except for the legacy framework)? We can consider dropping support for all projects. I just kept it because there were some preprocessor directives we used in the past, but I think we have already changed them. I need to double-check.

src/Testcontainers.Qdrant/QdrantBuilder.cs

This commit moves the wait for strategy to use the readyz endpoint. If a certificate has been specified, perform the check with TLS and allow any certificate to pass validation.

The qdrant gRPC client uses .NET Framework build of Grpc.Net.Client, which uses WinHttpHandler. The wiring up is easier on net462 if net462 is targeted specifically. Same reason why the Milvus container also targets net462 (testcontainers#1131 (comment)).

HofmeisterAn

Thank you for the PR 🙏. I have reviewed it and made some minor changes to align with the repository standards. I have a some suggestions regarding the certificates. I am happy to merge the PR after we have sorted them out.

HofmeisterAn · 2024-04-07T07:15:18Z

src/Testcontainers.Qdrant/QdrantConfiguration.cs

+	/// </summary>
+	public QdrantConfiguration(string apiKey = null, string certificate = null, string privateKey = null)
+	{
+		ApiKey = apiKey;


It looks like that the API key is never used anywhere afterward, you do not need to store it in the configuration.

HofmeisterAn · 2024-04-07T07:16:26Z

src/Testcontainers.Qdrant/QdrantBuilder.cs

+		var waitStrategy = Wait.ForUnixContainer().UntilHttpRequestIsSucceeded(request =>
+		{
+			var httpWaitStrategy = request.ForPort(QdrantHttpPort).ForPath("/readyz");
+
+			// allow any certificate defined to pass validation
+			if (DockerResourceConfiguration.Certificate is not null)
+			{
+				httpWaitStrategy.UsingTls()
+					.UsingHttpMessageHandler(new HttpClientHandler
+					{
+						ServerCertificateCustomValidationCallback = (_, _, _, _) => true 
+					});
+			}
+
+			return httpWaitStrategy;
+		});


Since we know the certificate, shouldn't we then check for the certificate? Or do we ignore it out of simplicity?

Was ignoring it out of simplicity for the waiting strategy, but it could be checked

HofmeisterAn · 2024-04-07T07:24:07Z

src/Testcontainers.Qdrant/Testcontainers.Qdrant.csproj

@@ -0,0 +1,12 @@
+<Project Sdk="Microsoft.NET.Sdk">
+    <PropertyGroup>
+        <TargetFrameworks>net6.0;net8.0;netstandard2.0;netstandard2.1</TargetFrameworks>


I think we should stick to the repo convention too 👍. In the end, it does not really matter, right (except for the legacy framework)? We can consider dropping support for all projects. I just kept it because there were some preprocessor directives we used in the past, but I think we have already changed them. I need to double-check.

tests/Testcontainers.Qdrant.Tests/X509CertificateGenerator.cs

…dotnet into qdrant-container

russcam added 5 commits March 30, 2024 09:17

Remove superfluous solution items

0181b8e

PR feedback

dc5f0cb

remove superfluous whitespace

93494d7

russcam mentioned this pull request Apr 2, 2024

Add Qdrant container #994

Closed

address PR checks

cb55a2f

russcam changed the title ~~Qdrant container~~ feat: add Qdrant module Apr 4, 2024

eerhardt reviewed Apr 4, 2024

View reviewed changes

eddumelendez reviewed Apr 6, 2024

View reviewed changes

src/Testcontainers.Qdrant/QdrantBuilder.cs Outdated Show resolved Hide resolved

russcam and others added 6 commits April 6, 2024 16:28

Use readyz endpoint

ca0b39b

This commit moves the wait for strategy to use the readyz endpoint. If a certificate has been specified, perform the check with TLS and allow any certificate to pass validation.

Also target net462

da10ea1

The qdrant gRPC client uses .NET Framework build of Grpc.Net.Client, which uses WinHttpHandler. The wiring up is easier on net462 if net462 is targeted specifically. Same reason why the Milvus container also targets net462 (testcontainers#1131 (comment)).

chore: Order csproj in sln

bb42811

chore: Replace SolutionDir with relative path

21a3439

chore: Remove BOM

f662ec6

chore: Fix minor repo inconsistencies

331abc5

HofmeisterAn reviewed Apr 7, 2024

View reviewed changes

russcam added 3 commits April 8, 2024 21:14

Use CertificateRequest to generate PEM certificate

96be0c7

Merge branch 'qdrant-container' of github.com:russcam/testcontainers-…

6b5e170

…dotnet into qdrant-container

remove BOM

f31250e

russcam requested a review from HofmeisterAn April 21, 2024 08:03

HofmeisterAn added enhancement New feature or request module An official Testcontainers module labels May 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Qdrant module #1149

feat: add Qdrant module #1149

russcam commented Apr 2, 2024 •

edited

netlify bot commented Apr 2, 2024 •

edited

eerhardt Apr 4, 2024

russcam Apr 6, 2024

HofmeisterAn Apr 7, 2024

HofmeisterAn left a comment

HofmeisterAn Apr 7, 2024

HofmeisterAn Apr 7, 2024

russcam Apr 8, 2024

HofmeisterAn Apr 7, 2024

	<TargetFrameworks>net6.0;net8.0;netstandard2.0;netstandard2.1</TargetFrameworks>
	<TargetFrameworks>net6.0;netstandard2.0</TargetFrameworks>

feat: add Qdrant module #1149

Are you sure you want to change the base?

feat: add Qdrant module #1149

Conversation

russcam commented Apr 2, 2024 • edited

What does this PR do?

Why is it important?

Related issues

How to test this PR

netlify bot commented Apr 2, 2024 • edited

✅ Deploy Preview for testcontainers-dotnet ready!

eerhardt Apr 4, 2024

Choose a reason for hiding this comment

russcam Apr 6, 2024

Choose a reason for hiding this comment

HofmeisterAn Apr 7, 2024

Choose a reason for hiding this comment

HofmeisterAn left a comment

Choose a reason for hiding this comment

HofmeisterAn Apr 7, 2024

Choose a reason for hiding this comment

HofmeisterAn Apr 7, 2024

Choose a reason for hiding this comment

russcam Apr 8, 2024

Choose a reason for hiding this comment

HofmeisterAn Apr 7, 2024

Choose a reason for hiding this comment

russcam commented Apr 2, 2024 •

edited

netlify bot commented Apr 2, 2024 •

edited