[Bug]: Unable to locate Dockerfile on v3.8.0

[davist-ir]

Testcontainers version

3.8.0

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

x64

.NET version

6.0.27

Docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:02 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.27.2 (137060)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 15.59GiB
 Name: docker-desktop
 ID: 62c93177-5541-478a-befa-219193e391a1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

What happened?

As part of our test projects with Testcontainers, we build the Docker image from the Dockerfile of the service in the solution. This has been working since a feature was added to Testcontainers to read the Dockerfile and pull down dependent images in the multistage build.

See below for an example Dockerfile:

FROM ghcr.io/[My Organization]/dotnet/aspnet:6.0 AS base
ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM ghcr.io/[My Organization]/dotnet/sdk:6.0 AS build
ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY ["src/Api/Api.csproj", "src/Api/"]
RUN dotnet restore "./src/Api/Api.csproj"
COPY . .
WORKDIR "/src/src/Api"
RUN dotnet build "..Api.csproj" -c $BUILD_CONFIGURATION -o /app/build

FROM build AS publish
ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish "./Api.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false

FROM base AS final
ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
LABEL "org.opencontainers.image.source"=https://github.com/[My Organization]/Api
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Api.dll"]

Example of the IImage implementation:

internal class ApiImage
    : IImage,
    IAsyncLifetime
{
    public string Repository
        => this._image.Repository;

    public string Name
        => this._image.Name;

    public string Tag
        => this._image.Tag;

    public string FullName
    => this._image.FullName;

    private readonly IImage _image = new DockerImage("ghcr.io/[My Organization]", "api", DateTimeOffset.UtcNow.ToUnixTimeSeconds().ToString());

    private readonly SemaphoreSlim _semaphoreSlim = new(1, 1);

    public Task DisposeAsync()
        => Task.CompletedTask;

    public string GetHostname()
        => this._image.GetHostname();

    public async Task InitializeAsync()
    {
        await this._semaphoreSlim.WaitAsync();

        try
        {
            await new ImageFromDockerfileBuilder()
                .WithName(this)
                .WithDeleteIfExists(false)
                .WithDockerfile("./src/Api/Dockerfile")
                .WithDockerfileDirectory(CommonDirectoryPath.GetSolutionDirectory(), string.Empty)
                .WithBuildArgument("RESOURCE_REAPER_SESSION_ID", ResourceReaper.DefaultSessionId.ToString("D"))
                .Build()
                .CreateAsync();
        }
        finally
        {
            this._semaphoreSlim.Release();
        }
    }
}

This has not changes for a bit now, but as we are upgrading from v3.7.0 to v3.8.0, we are experiencing an issue. See below for the exception that is thrown when running the tests.

Error Message:
   System.AggregateException : One or more errors occurred. (One or more errors occurred. (Docker API responded with status code=InternalServerError, response={"message":"Cannot locate specified Dockerfile: ./src/Api/Dockerfile"}
)) (The following constructor parameters did not have matching fixture data: ApiFixture fixture)
---- System.AggregateException : One or more errors occurred. (Docker API responded with status code=InternalServerError, response={"message":"Cannot locate specified Dockerfile: ./src/Api/Dockerfile"}
)
-------- Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"message":"Cannot locate specified Dockerfile: ./src/Api/Dockerfile"}

---- The following constructor parameters did not have matching fixture data:ApiFixture fixture
  Stack Trace:

----- Inner Stack Trace #1 (System.AggregateException) -----
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Api.IntegrationTests.Bootstrap.Fixtures.ApiFixture..ctor() in C:\Workspace\Api\tests\Api.IntegrationTests\Bootstrap\Fixtures\ApiFixture.cs:line 22
----- Inner Stack Trace -----
   at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response)
   at Docker.DotNet.DockerClient.MakeRequestForRawResponseAsync(HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, CancellationToken token)
   at Docker.DotNet.Models.StreamUtil.MonitorResponseForMessagesAsync[T](Task`1 responseTask, DockerClient client, CancellationToken cancel, IProgress`1 progress)
   at DotNet.Testcontainers.Clients.DockerImageOperations.BuildAsync(IImageFromDockerfileConfiguration configuration, ITarArchive dockerfileArchive, CancellationToken ct) in /_/src/Testcontainers/Clients/DockerImageOperations.cs:line 122
   at DotNet.Testcontainers.Clients.TestcontainersClient.BuildAsync(IImageFromDockerfileConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 349
   at DotNet.Testcontainers.Images.FutureDockerImage.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Images/FutureDockerImage.cs:line 116
   at DotNet.Testcontainers.Images.FutureDockerImage.CreateAsync(CancellationToken ct) in /_/src/Testcontainers/Images/FutureDockerImage.cs:line 82
   at Api.IntegrationTests.Bootstrap.Images.ApiImage.InitializeAsync() in C:\Workspace\Api\tests\Api.IntegrationTests\Bootstrap\Images\ApiImage.cs:line 39
   at Api.IntegrationTests.Bootstrap.TestContainers.InitializeAsync() in C:\Workspace\Api\tests\Api.IntegrationTests\Bootstrap\Containers\TestContainers.cs:line 84
----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

System.AggregateException : One or more errors occurred. (Docker API responded with status code=InternalServerError, response={"message":"Cannot locate specified Dockerfile: ./src/Api/Dockerfile"}

This seems to be the root of the issue. When Testcontainers goes to create the image, it is unable to locate the Dockerfile. It seems the code that is actually throwing the issue is in Docker.DotNet package, but I'm not sure if it is an issue in there or Testcontainers.DotNet.

Expected Behavior

The excepted behavior is to locate the Dockerfile, read it and see that it has dependent images, pull the dependent images, and then build the image using the Dockerfile. This is how it works in v3.7.0.

Relevant log output

No response

Additional information

No response

[HofmeisterAn]

Hm, this is unfortunate. I am not running into this issue in my demos or tests. We need more information to figure out what is going wrong.

Testcontainers creates a tarball of the Dockerfile directory and sends it to the daemon. This tarball must contain the Dockerfile src/Api/Dockerfile. I guess this file is somehow missing. Right now, I believe it might be an entry in the .dockerignore file that is causing the issue. Can you please double-check the .dockerignore file and make sure the Dockerfile is not ignored (maybe temporarily remove the .dockerignore file)?

If that does not help, we need to look into the tarball. Set a breakpoint here or stop at the exception and navigate to the tarball in the %TEMP% directory (the file stream will contain the actual path) and check its content. Make sure the Dockerfile is stored in the correct path.

[davist-ir]

@HofmeisterAn This is interesting because I have been having issues with Testcontainers running in VS recently, but I haven't been able to identify the root cause. The error I get is something about the library not being able to write a "tar archive". I was never able to figure out a difference with it between versions and the tests work completely find from my terminal running dotnet test, which is also how our CI/CD pipeline works. I will try to put together a minimal reproduction of the issue and share it here. I will also check our .dockerignore and see if that is causing the issue. It seems odd that would be the cause because the .dockerignore is the same between when I run the tests with v.3.7.0 and v.3.8.0. Anyways, I will see what I can put together and either share the repo with the minimal reproduction or report back that I found the issue.

[HofmeisterAn]

It seems odd that would be the cause because the .dockerignore is the same between when I run the tests with v.3.7.0 and v.3.8.0.

The previous version contained a bug; 3.8.0 addresses it (#1122).

[davist-ir]

@HofmeisterAn https://github.com/davist-ir/Sandbox Here is a reproducible solution. There is a v3.7.0 branch that works fine from the terminal using dotnet test and VS. There is a v3.8.0 branch that doesn't work for either.

Some things I noticed while putting this together and testing what you said originally. There is an entry in the .dockerignore that says, **/Dockerfile*. If I remove that line, both versions work. Meaning I can get the tests to run fine on v3.7.0 with no issues and no changes needed. When I move to v.3.8.0, I have to remove that entry from the .dockerignore file. FWIW this is the .dockerignore file produced by VS after selecting to "Add Docker Support" to a project and has been unchanged from various solutions we have for a while now.

[HofmeisterAn]

There is an entry in the .dockerignore file that says, **/Dockerfile*. If I remove that line, both versions work.

The Dockerfile must be included in the tarball and cannot be ignored.

You can use the .dockerignore file to exclude the Dockerfile and .dockerignore files. However, these files are still sent to the builder as they are needed for running the build.

Testcontainers does this too. We make sure that the Dockerfile passed to the builder is included as well. I guess the ./ breaks it though in combination with the mentioned fix above. Can you simply try src/Api/Dockerfile?

[davist-ir]

The Dockerfile must be included in the tarball and cannot be ignored.

This makes sense to me. I was actually surprised to find the **/Dockerfile* in the .dockerignore because it is the one generated by VS when using the "Add Docker Support" feature.

Can you simply try src/Api/Dockerfile?

I can. Here is the PR for the repo. Making that change does seem to fix the issue. davist-ir/Sandbox#1

[HofmeisterAn]

Making that change does seem to fix the issue. davist-ir/Sandbox#1

Probably the simplest thing we can do is to add the negate pattern twice, with and without the preceding ./:

[davist-ir]

@HofmeisterAn I'm not going to pretend I fully understand the ins and outs of the Testcontainers inner workings, but I was stepping through the code yesterday to find where the root of the issue would be. I came across this line:

testcontainers-dotnet/src/Testcontainers/Images/DockerfileArchive.cs

Line 139 in 533a249

var relativeFilePath = absoluteFilePath.Substring(dockerfileDirectoryPath.TrimEnd(Path.AltDirectorySeparatorChar).Length + 1);

It grabs the "relative path" from the absolute path and then uses that to compare/match with the Regex from the IgnoreFile class. The issue becomes that the "relative path" it pulls out does not contain the ./, while the IgnorFile patterns take the exact value passed to it. In my case, I provided ./src/Sandbox.Api/Dockerfile, but the relativePath variable is simply src/Sandbox.Api/Dockerfile. I'm not sure I see how adding the negate pattern twice, with and without the preceding ./ would address that issue.

With that said, for now, I am simply changing the string for the Dockerfile as src/Sandbox.Api/Dockerfile and moving on. I'm not sure the best way to fix the issue, but since I have a workaround, I can at least progress. I appreciate you helping debug and figure out the issue.

[HofmeisterAn]

The issue becomes that the "relative path" it pulls out does not contain the ./, while the IgnorFile patterns take the exact value passed to it. In my case, I provided ./src/Sandbox.Api/Dockerfile, but the relativePath variable is simply src/Sandbox.Api/Dockerfile.

👍 Ah, then it is the other way around.

I'm not sure I see how adding the negate pattern twice, with and without the preceding ./ would address that issue.

Not sure if I am missing something, but this will ensure that the Dockerfile will definitely be included, in case it gets excluded by "accident" from a previous ignore entry. But I agree, it is difficult to say if that is the appropriate fix. Thanks for the further investigation. That helps a lot to finally sort it out.

I appreciate you helping debug and figure out the issue.

Happy to help 👍.