copyFileFromContainer returns an empty file if file permissions inside docker container don't have w for owner user

[xtrm0]

Whenever I copyFileFromContainer I always get an empty file, even though the same command using docker cp executes correctly.

I'm not sure how to provide more information, so here goes my configuration:

I have a docker image with the following start script, that generates the file server_pub.pem:

set -e
pd=`pwd`
cd src/main/resources/
openssl ecparam -name secp256k1 -genkey -noout -out server_prv.pem
openssl ec -in server_prv.pem -pubout -out server_pub.pem
cd $pd
java -jar server.jar

And when I run outside this image the copyFileFromContainer I get an empty file:

server = new GenericContainer("gennsecserver:latest").withExposedPorts(8080).waitingFor(Wait.forHttp("/api/ping"));
server.start();
Path tempDirectory = Files.createTempDirectory("");
String targetPubKeyPath = tempDirectory.toAbsolutePath().toString() + "/server_pub.pem";
String baseurl = "http://" + server.getContainerIpAddress() + ":" + server.getMappedPort(8080);
server.copyFileFromContainer("/home/secserv/src/main/resources/server_pub.pem", targetPubKeyPath);

But if I run docker cp I get a valid file:

docker cp 5dea99a39571:/home/secserv/src/main/resources/server_pub.pem ./

[xtrm0]

Ok. I found the reason. If the file is created with r only permissions the library will first create the file and give it their atributes, and once it tries to write to the file, it will fail. This is a bug. For now, this is my solution: replace server.copyFileFromContainer with:

String cmd = "docker cp " + container.getContainerId() + ":" + file + " " + destFile;
java.lang.Runtime.getRuntime().exec(cmd);

[kiview]

Hi @xtrm0, thanks for providing this information.
Would it be possible for you to add an example project, demonstrating this bug?

[xtrm0]

This was my docker file for gensecserver:

#This image is for a docker server that generates an keypair upon creation
FROM openjdk:8
RUN rm /bin/sh && ln -s /bin/bash /bin/sh

RUN mkdir -p /home/secserv/

RUN mkdir -p /home/secserv/src/main/resources/

COPY application.yml /home/secserv/src/main/resources/application.yml
COPY genKeys.sh /home/secserv/src/main/resources/genKeys.sh
COPY starter.sh /home/secserv/starter.sh

RUN chmod +x /home/secserv/src/main/resources/genKeys.sh && chmod +x /home/secserv/starter.sh

WORKDIR /home/secserv
CMD /home/secserv/starter.sh
EXPOSE 8080

COPY server.jar /home/secserv/

This was starter.:

set -e
pd=`pwd`
cd src/main/resources/
./genKeys.sh
cd $pd
java -jar server.jar

And this the genKeys.sh:

set -e
if [ ! -f server_prv.pem ]; then
  openssl ecparam -name secp256k1 -genkey -noout -out server_pair.pem
  openssl ec -in server_pair.pem -pubout -out server_pub.pem
  openssl ec -in server_pair.pem -no_public -out server_preprv.pem
  openssl pkcs8 -topk8 -inform pem -in server_preprv.pem -outform pem -nocrypt -out server_prv.pem
  rm server_preprv.pem
  rm server_pair.pem
fi

The server.jar is irrelevant here as it was not used. So you can just replace that with any command that will keep running, like for instance "/usr/bin/sleep 99999999".

[xtrm0]

In short, this should reproduce the bug:

#This image is for a docker server that generates an keypair upon creation
FROM openjdk:8
RUN rm /bin/sh && ln -s /bin/bash /bin/sh

RUN mkdir -p /home/secserv/

COPY starter.sh /home/secserv/starter.sh

RUN chmod +x /home/secserv/starter.sh

WORKDIR /home/secserv
CMD /home/secserv/starter.sh
EXPOSE 8080

starter.sh:

set -e
if [ ! -f server_prv.pem ]; then
  openssl ecparam -name secp256k1 -genkey -noout -out server_pair.pem
  openssl ec -in server_pair.pem -pubout -out server_pub.pem
  openssl ec -in server_pair.pem -no_public -out server_preprv.pem
  openssl pkcs8 -topk8 -inform pem -in server_preprv.pem -outform pem -nocrypt -out server_prv.pem
  rm server_preprv.pem
  rm server_pair.pem
fi
sleep 99999

java to run on host:

server = new GenericContainer("gennsecserver:latest");
server.start();
//Sleep here? for a few seconds maybe, to make sure that the problem is not with filesystem updates
Path tempDirectory = Files.createTempDirectory("");
String targetPubKeyPath = tempDirectory.toAbsolutePath().toString() + "/server_pub.pem";
server.copyFileFromContainer("/home/secserv/src/main/resources/server_pub.pem", targetPubKeyPath);

[beargiles]

Someone on our team is reporting the same problem. I can successfully copy a file from the container on my linux machine but the same code is failing on their macs.

I haven't looked at the code so this may be totally wrong but is it possible that the code is doing:

1. create output file
2. set permissions
3. write contents

instead of

1. create output file
2. write contents
3. set permissions

IIRC on Linux the order won't matter since you retain the permissions from when the file was opened. That's why one common trick for getting a huge buffer is to open a temporary file and immediately delete it. It's still available to the application but is inaccessible by anyone else. Changing the file permissions so it's read-only won't affect the application until the file is closed.

However Macs may be different - they might check the permissions on every access and changing the permissions immediately would block future writes. It's more secure but it breaks this technique for copying a file.

That's pure conjecture though since I don't know if that's the actual behavior on macs or if the code is actually doing the file write and changing permissions in that order.

[beargiles]

A follow-up - my coworkers stepped through it with a debugger. It looks like it's a known issue in the Apache commons 'compress' library that was recently fixed. We're checking which version of the library we're actually using once all of the dependencies are resolved.

The details was that it's a 'bad header' error in the tar archiver. They were confused by that until I pointed out that a docker image is a stack of tar files. I hope that info helps other people who see this or similar errors.

[rnorth]

We believe this is all down to an issue which we've inherited from docker-java, docker-java/docker-java#1079. This affects the docker-java netty implementation.

I'm testing a fix in the form of #817, which seems to be working well so far.

There is a workaround which works without #817: we have an alternative OkHttp transport that is already bundled with Testcontainers, as a future replacement for netty. The OkHttp transport is immune to this bug, so may be a useful interim workaround. To activate it, add the line transport.type=okhttp to ~/.testcontainers.properties.

[rnorth]

We've released 1.8.3 which hopefully fixes this issue. Please could you try it out?

[beargiles]

It works for us.

…

On Sun, Aug 5, 2018 at 3:37 PM, Richard North ***@***.***> wrote: We've released 1.8.3 <https://github.com/testcontainers/testcontainers-java/releases/tag/1.8.3> which hopefully fixes this issue. Please could you try it out? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#681 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMqTOiD8UqNLvNCIiR4-yK2u08J9TTsks5uN2WogaJpZM4Tx4dM> .