x509: certificate signed by unknown authority

[davidzhongsydney]

go install in docker image report x509: certificate signed by unknown

What versions are you running?

uname -r

5.4.72-microsoft-standard-WSL2

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ go list -m github.com/chromedp/chromedp
go: github.com/chromedp/chromedp@v0.6.10: missing go.sum entry; to add it:
go mod download github.com/chromedp/chromedp

$ google-chrome --version
sh: 137: google-chrome: not found

$ go version
go version go1.16.5 linux/amd64

What did you do? Include clear steps.

docker pull chromedp/headless-shell:74.0.3717.1
//download go1.16.5.linux-amd64.tar.gz and put into local /goinstall directoy
docker run -v ${PWD}/goinstall/:/tmp/ -d --name chromedpcontainer chromedp/headless-shell

docker exec -it sh
cd /tmp
tar -C /usr/local -xzf go1.16.5.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
cd /tmp
go mod init helloworld
go get -x github.com/chromedp/chromedp

What did you expect to see?

github.com/chromedp/chromedp installed. below added into go.mod

require (
github.com/chromedp/chromedp v0.6.10
)

What did you see instead?

go: github.com/chromedp/chromedp@v0.6.10: Get "https://proxy.golang.org/github.com/chromedp/chromedp/@v/v0.6.10.mod": x509: certificate signed by unknown authority

[ghost]

git clean

Remove untracked files from the working tree

[ghost]

git bisect

Use binary search to find the commit that introduced a bug

[kenshaw]

@Oneartwe I have no idea what this is from or for, but your issue above looks like an issue with Go, and nothing to do with chromedp. Closing this. Please reopen if I'm not understanding something. Thanks.

[ZekeLu]

@davidzhongsydney I don't know your use case, but generally speaking, it's not a good idea to build go app using the chromedp/headless-shell image. The recommended way is to use multi-stage builds. That is, you build the go app using the golang image first, then copy the output to the chromedp/headless-shell image.

[davidzhongsydney]

@ZekeLu I know how to pull and run from golang image. But I am not sure how "copy the output to the chromedp/headless-shell image". Would you be able to show me how the docker command looks like please? Thanks.

[davidzhongsydney]

@kenshaw I also tried on ubuntu image. Manually install golang and test the helloworld. It is all good.
But when I run: go get -x github.com/chromedp/chromedp. It return same error "x509: certificate signed by unknown authority"

docker run -it --name ubuntucontainer ubuntu

[davidzhongsydney]

go get -x -insecure github.com/chromedp/chromedp command works fine under my Windows development environment. Does the Linux docker image missing the CA certificate?

[davidzhongsydney]

@ZekeLu As you advised, I used golang:alpine image, [ go get -x github.com/chromedp/chromedp] is execuated without error. But chromedp by default will use headless as bowser. But does not installed.

As you mentioned, "copy the output to the chromedp/headless-shell image". Could you please let me know how to copy the headless to the golang:alpine image? Thanks.

2021/07/11 04:52:58 exec: "google-chrome": executable file not found in $PATH

[ZekeLu]

Here is an example Dockerfile copied from the docker document:

FROM golang:1.16
WORKDIR /go/src/github.com/alexellis/href-counter/
RUN go get -d -v golang.org/x/net/html
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=0 /go/src/github.com/alexellis/href-counter/app .
CMD ["./app"]

Please read https://docs.docker.com/develop/develop-images/multistage-build/ for more information.

[davidzhongsydney]

@ZekeLu Thanks for your advice.

I used the method you recommended. Docker image can be successfully created. However, when I run the docker image, it report "Failed to load GLES library".

FROM golang:alpine
WORKDIR /app
COPY go.mod .
COPY go.sum . 
RUN go mod download
COPY . .
RUN go build -o chromedptest

FROM chromedp/headless-shell:74.0.3717.1
WORKDIR /apps
COPY --from=0 /app/chromedptest .
CMD ["./chromedptest"]

PS C:\Users\weizh> docker run -it chromedpimg
[0712/142732.047364:WARNING:resource_bundle.cc(357)] locale_file_path.empty() for locale

DevTools listening on ws://0.0.0.0:9222/devtools/browser/7c9742ad-ea03-4592-85a9-0ac9689fd900
[0712/142732.734765:WARNING:resource_bundle.cc(357)] locale_file_path.empty() for locale
[0712/142732.814373:ERROR:egl_util.cc(60)] Failed to load GLES library: /headless-shell/swiftshader/libGLESv2.so: /headless-shell/swiftshader/libGLESv2.so: cannot open shared object file: No such file or directory
[0712/142732.887671:ERROR:viz_main_impl.cc(170)] Exiting GPU process due to errors during initialization

Then I run the image directly from https://github.com/chromedp/docker-headless-shell. It also report error. Is the image itself has problem?

PS E:\Go\Chromedp_test3> docker run -d -p 9222:9222 --name headless-shell chromedp/headless-shell
59c831ed158d7b841d1ac311dfb8f50f0c1fb34cd3b04b55103a97f47d7568f9

PS E:\Go\Chromedp_test3> docker logs 59c
[0712/144112.081756:WARNING:resource_bundle.cc(405)] locale_file_path.empty() for locale

DevTools listening on ws://0.0.0.0:9222/devtools/browser/668ddd91-829c-4ba6-b251-41aec7293eab
[0712/144112.899336:WARNING:resource_bundle.cc(405)] locale_file_path.empty() for locale
[0712/144113.342103:ERROR:gpu_init.cc(440)] Passthrough is not supported, GL is swiftshader

[davidzhongsydney]

2021/07/13 05:58:32 could not dial "ws://127.0.0.1:44613/devtools/browser/4798c484-90f7-431b-be8c-f2c0474547f4": dial tcp 127.0.0.1:44613: connect: connection refused
exit status 1

[davidzhongsydney]

the root cause of starting the chromedp/headless-shell image is there is no libraries below.
/headless-shell/swiftshader/libGLESv2.so
/headless-shell/swiftshader/libEGL.so

However, these two lib can be found under /headless-shell/. After mkdir /headless-shell/swiftshader and copy the so files into dir, and retart the container. Error is resolved.
/headless-shell/libGLESv2.so
/headless-shell/libEGL.so

DevTools listening on ws://0.0.0.0:9222/devtools/browser/8811166a-958b-4390-9b2a-3ea799746744
[0713/072600.864799:WARNING:resource_bundle.cc(357)] locale_file_path.empty() for locale

[ZekeLu]

@davidzhongsydney Good job!

But please note that chromedp/headless-shell 74.0.3717.1 is too old. The recent versions of chromedp/headless-shell do have libEGL.so and libEGLv2.so in /headless-shell/swiftshader/.

When you run docker run -d -p 9222:9222 --name headless-shell chromedp/headless-shell without specifying the version, it will use the latest version, which is most likely not the same as 74.0.3717.1. That's why the outputs are differenct.

When I run chromedp/headless-shell:91.0.4472.114, I get errors in the output too:

[0713/085812.938349:WARNING:resource_bundle.cc(405)] locale_file_path.empty() for locale 
[0713/085812.951583:WARNING:resource_bundle.cc(405)] locale_file_path.empty() for locale 
[0713/085812.952043:WARNING:resource_bundle.cc(405)] locale_file_path.empty() for locale 

DevTools listening on ws://0.0.0.0:9222/devtools/browser/cc3c2a1f-29a7-40fc-9303-141f4e8e5a8a
[0713/085812.983498:ERROR:gpu_init.cc(440)] Passthrough is not supported, GL is swiftshader
[0713/085812.989169:WARNING:resource_bundle.cc(405)] locale_file_path.empty() for locale 

But this is not a blocking error. I have used it in production environment without any issue.

[davidzhongsydney]

@ZekeLu It works now. Thanks.

[davidzhongsydney]

@ZekeLu each time when the container restarted, it listen on different url.
eg. ws://0.0.0.0:9222/devtools/browser/8811166a-958b-4390-9b2a-3ea799746744

How did you program get this dynamic url? Is there some way to make the URL static? Thanks.

[kenshaw]

If you manually launch chrome, you need to specify the port it listens on. I believe the modern versions of chrome choose a random port. You can specify this via command line options, which can be used with the RemoteAllocator. Please see here: http://peter.sh/experiments/chromium-command-line-switches/

[ZekeLu]

Yes, as @kenshaw pointed out, you could use the RemoteAllocator, and it can query the correct endpoint for you. Please note that this is a new feature introduced in v0.7.3 (by #817). You can check the PR for an example.

[ZekeLu]

How did you program get this dynamic url? Is there some way to make the URL static?

Oh, forgot to answer this question. I don't think we can make the URL static. For now, the ExecAllocator will read the dynamic URL from the process output:

chromedp/allocate.go

Lines 275 to 279 in e43a192

	// readOutput grabs the websocket address from chrome's output, returning as
	// soon as it is found. All read output is forwarded to forward, if non-nil.
	// done is used to signal that the asynchronous io.Copy is done, if any.
	func readOutput(rc io.ReadCloser, forward io.Writer, done func()) (wsURL string, _ error) {
	prefix := []byte("DevTools listening on")

And RemoteAllocator will query the URL from the endpoint http://[IP]:[port]/json/version

chromedp/util.go

Lines 35 to 43 in e43a192

	// detectURL detects the websocket debugger URL if the provided URL is not a
	// valid websocket debugger URL.
	//
	// A valid websocket debugger URL is something like:
	// ws://127.0.0.1:9222/devtools/browser/...
	// The original URL with the following formats are accepted:
	// * ws://127.0.0.1:9222/
	// * http://127.0.0.1:9222/
	func detectURL(urlstr string) string {

[davidzhongsydney]

@ZekeLu @kenshaw Yeah. I am using NewRemoteAllocator as you recommended. It works. 👍

        var devToolWsUrl ="ws://0.0.0.0:9222/devtools/browser/5272c7b9-6c08-4dc7-bbcc-49b6129cb24b"
	allocCtx, cancel := chromedp.NewRemoteAllocator(context.Background(), devToolWsUrl)
	defer cancel()

[davidzhongsydney]

em~~~back to my previous question. Since the URL that headless listen on is unknow until it is running in the container, how the golang program can capture this URL?

Right now what I did is

1. run the image

docker run  -d --name chromedpcontainer chromedp/headless-shell

2. get the listen url by checking the container log

docker logs <container id>

3. get into the image

docker exec -it <container id> sh

4. edit the url, then run the golang

It would be better if the golang can capture the listen URL rather than being hardcoded

[ZekeLu]

@davidzhongsydney It seems that you missed this comment #871 (comment). Please do read #817.

[davidzhongsydney]

@ZekeLu Great! My problem is resolved.

[davidzhongsydney]

@ZekeLu Thanks very much.