Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker requested access to the resource is denied #1180

Open
ravi1216 opened this issue Jul 3, 2018 · 9 comments
Open

Docker requested access to the resource is denied #1180

ravi1216 opened this issue Jul 3, 2018 · 9 comments
Labels
area/builder area/distribution version/1.13

Comments

@ravi1216
Copy link

ravi1216 commented Jul 3, 2018 •
edited by cpuguy83

I am not able to Push Docker images into the docker hub after login.

When I am pushing the Image following is my output

The push refers to repository [docker.io/yadaraviteja/hellonode]
fc5e6e40c850: Preparing
67f1318aa14f: Preparing
1868b3bf7855: Preparing
2895be281ac1: Preparing
ab90d83fa34a: Preparing
8ee318e54723: Preparing
e6695624484e: Preparing
da59b99bbd3b: Preparing
5616a6292c16: Preparing
f3ed6cb59ab0: Preparing
654f45ecb7e3: Preparing
2c40c66f7667: Preparing
8ee318e54723: Waiting
e6695624484e: Waiting
da59b99bbd3b: Waiting
5616a6292c16: Waiting
f3ed6cb59ab0: Waiting
654f45ecb7e3: Waiting
2c40c66f7667: Waiting
denied: requested access to the resource is denied
Build step 'Docker Build and Publish' marked build as failure
Finished: FAILURE
I am using the docker container for the Jenkins UP and running. Following is my DockerFile 
FROM openshift/origin

# Jenkins image for OpenShift
#
# This image provides a Jenkins server, primarily intended for integration with
# OpenShift v3.
#
# Volumes:
# * /var/jenkins_home
# Environment:
# * $JENKINS_PASSWORD - Password for the Jenkins 'admin' user.

MAINTAINER Ben Parees <bparees@redhat.com>

# Jenkins LTS packages from
# https://pkg.jenkins.io/redhat-stable/
ENV JENKINS_VERSION=2 \
    HOME=/var/lib/jenkins \
    JENKINS_HOME=/var/lib/jenkins \
    JENKINS_UC=https://updates.jenkins.io \
    OPENSHIFT_JENKINS_IMAGE_VERSION=3.11 \
    LANG=en_US.UTF-8 \
    LC_ALL=en_US.UTF-8

LABEL k8s.io.description="Jenkins is a continuous integration server" \
      k8s.io.display-name="Jenkins 2" \
      openshift.io.expose-services="8080:http" \
      openshift.io.tags="jenkins,jenkins2,ci" \
      io.openshift.s2i.scripts-url=image:///usr/libexec/s2i

# 8080 for main web interface, 50000 for slave agents
EXPOSE 8080 50000

RUN curl https://pkg.jenkins.io/redhat-stable/jenkins.repo -o /etc/yum.repos.d/jenkins.repo && \
    rpm --import https://pkg.jenkins.io/redhat-stable/jenkins-ci.org.key && \
    yum install -y centos-release-scl-rh && \
    curl https://copr.fedorainfracloud.org/coprs/alsadi/dumb-init/repo/epel-7/alsadi-dumb-init-epel-7.repo -o /etc/yum.repos.d/alsadi-dumb-init-epel-7.repo && \
    x86_EXTRA_RPMS=$(if [ "$(uname -m)" == "x86_64" ]; then echo -n java-1.8.0-openjdk.i686 java-1.8.0-openjdk-devel.i686 ; fi) && \
    INSTALL_PKGS="dejavu-sans-fonts rsync gettext git tar zip unzip openssl bzip2 dumb-init java-1.8.0-openjdk jenkins-2.107.3-1.1" && \
    yum -y --setopt=tsflags=nodocs install $INSTALL_PKGS $x86_EXTRA_RPMS && \
    # have temporarily removed the validation for java to work around known problem fixed in fedora; jupierce and gmontero are working with
    # the requisit folks to get that addressed ... will switch back to rpm -V $INSTALL_PKGS when that occurs
    rpm -V  dejavu-sans-fonts rsync gettext git tar zip unzip openssl bzip2 dumb-init jenkins-2.107.3-1.1 && \
    yum clean all  && \
    localedef -f UTF-8 -i en_US en_US.UTF-8

COPY ./contrib/openshift /opt/openshift
COPY ./contrib/jenkins /usr/local/bin
ADD ./contrib/s2i /usr/libexec/s2i
ADD release.version /tmp/release.version

RUN /usr/local/bin/install-plugins.sh /opt/openshift/base-plugins.txt && \
    # need to create <plugin>.pinned files when upgrading "core" plugins like credentials or subversion that are bundled with the jenkins server
    # Currently jenkins v2 does not embed any plugins, but for reference:
    # touch /opt/openshift/plugins/credentials.jpi.pinned && \
    rmdir /var/log/jenkins && \
    chmod 664 /etc/passwd && \
    chmod -R 775 /etc/alternatives && \
    chmod -R 775 /var/lib/alternatives && \
    chmod -R 775 /usr/lib/jvm && \
    chmod 775 /usr/bin && \
    chmod 775 /usr/lib/jvm-exports && \
    chmod 775 /usr/share/man/man1 && \
    chmod 775 /var/lib/origin && \
    unlink /usr/bin/java && \
    unlink /usr/bin/jjs && \
    unlink /usr/bin/keytool && \
    unlink /usr/bin/orbd && \
    unlink /usr/bin/pack200 && \
    unlink /usr/bin/policytool && \
    unlink /usr/bin/rmid && \
    unlink /usr/bin/rmiregistry && \
    unlink /usr/bin/servertool && \
    unlink /usr/bin/tnameserv && \
    unlink /usr/bin/unpack200 && \
    unlink /usr/lib/jvm-exports/jre && \
    unlink /usr/share/man/man1/java.1.gz && \
    unlink /usr/share/man/man1/jjs.1.gz && \
    unlink /usr/share/man/man1/keytool.1.gz && \
    unlink /usr/share/man/man1/orbd.1.gz && \
    unlink /usr/share/man/man1/pack200.1.gz && \
    unlink /usr/share/man/man1/policytool.1.gz && \
    unlink /usr/share/man/man1/rmid.1.gz && \
    unlink /usr/share/man/man1/rmiregistry.1.gz && \
    unlink /usr/share/man/man1/servertool.1.gz && \
    unlink /usr/share/man/man1/tnameserv.1.gz && \
    unlink /usr/share/man/man1/unpack200.1.gz && \
    chown -R 1001:0 /opt/openshift && \
    /usr/local/bin/fix-permissions /opt/openshift && \
    /usr/local/bin/fix-permissions /opt/openshift/configuration/init.groovy.d && \
    /usr/local/bin/fix-permissions /var/lib/jenkins && \
    /usr/local/bin/fix-permissions /var/log

VOLUME ["/var/lib/jenkins"]

USER 1001
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
CMD ["/usr/libexec/s2i/run"]

My Jenkins is running perfect in the "ECS" and I am able to build the job successfully. From the ECS cluster EC2 instance I am able to push the image successfully. When I am going into the jenkins docker container by using "docker exec -it "my-images" /bin/bash". This container is using the EC2-instance host docker demon. Docker login also Success but I am not able to push the image into the repo from the jenkins container.

Steps to reproduce the issue:

1.[root@My-ip ~]# docker exec -it b022366b391a /bin/bash
2.bash-4.2$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: yadaraviteja
Password:
Login Succeeded
bash-4.2$ docker images
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
jenkins-m                 latest              6a1f4d41462f        32 minutes ago      1.76 GB
yadaraviteja/hellonode    latest              ee13ed64bf79        6 days ago          662 MB
hello                     latest              56af76c2c878        7 days ago          662 MB
yadaraviteja/hellonode    1                   56af76c2c878        7 days ago          662 MB
<none>                    <none>              1452091493c8        7 days ago          1.23 GB
redis                     latest              55cb7014c24f        12 days ago         83.4 MB
openshift/origin          latest              c45099c2bcfa        2 weeks ago         674 MB
jenkins                   latest              07b4164f9789        3 weeks ago         696 MB
yadaraviteja/hellonode    <none>              7452be918594        3 weeks ago         676 MB
centos                    7                   49f7960eb7e4        4 weeks ago         200 MB
centos                    latest              49f7960eb7e4        4 weeks ago         200 MB
amazon/amazon-ecs-agent   latest              622111e45fde        7 weeks ago         29.4 MB
node                      7-onbuild           3de424ca76d2        10 months ago       660 MB

3.docker tag ee13ed64bf79 yadaraviteja/hellonode:2
bash-4.2$ docker push yadaraviteja/hellonode:2
The push refers to a repository [docker.io/yadaraviteja/hellonode]

Describe the results you received:

9d834649f96c: Preparing
7864babea84b: Preparing
9ee59d11e6c2: Preparing
2895be281ac1: Preparing
ab90d83fa34a: Preparing
8ee318e54723: Waiting
e6695624484e: Waiting
da59b99bbd3b: Waiting
5616a6292c16: Waiting
f3ed6cb59ab0: Waiting
654f45ecb7e3: Waiting
2c40c66f7667: Waiting
denied: requested access to the resource is denied

Describe the results you expected:

Docker logs:

Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision
INFO: Asked to provision 1 slave(s) for: slave
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision
INFO: Will provision ECS Slave slave, for label: slave
Jul 03, 2018 3:02:55 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning ECS Slave slave from slave with 1 executors. Remaining excess workload: 0
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Found 2 instances
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Resources found in instance arn:aws:ecs:us-east-1:318896727843:container-instance/37552fa1-2ac8-4551-a2f2-ae6c177ffc7f: [{Name: CPU,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 512,StringSetValue: []}, {Name: MEMORY,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 1326,StringSetValue: []}, {Name: PORTS,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: [22, 2376, 2375, 51678, 51679, 49164]}, {Name: PORTS_UDP,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: []}]
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:us-east-1:318896727843:container-instance/37552fa1-2ac8-4551-a2f2-ae6c177ffc7f has 1,326mb of free memory. 2,048mb are required
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:us-east-1:318896727843:container-instance/37552fa1-2ac8-4551-a2f2-ae6c177ffc7f has 512 units of free cpu. 512 units are required
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Resources found in instance arn:aws:ecs:us-east-1:318896727843:container-instance/7ab12d80-8a0d-4b00-9525-7ed3b39aca46: [{Name: CPU,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 1536,StringSetValue: []}, {Name: MEMORY,Type: INTEGER,DoubleValue: 0.0,LongValue: 0,IntegerValue: 7467,StringSetValue: []}, {Name: PORTS,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: [22, 2376, 2375, 51678, 51679, 49164]}, {Name: PORTS_UDP,Type: STRINGSET,DoubleValue: 0.0,LongValue: 0,IntegerValue: 0,StringSetValue: []}]
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:us-east-1:318896727843:container-instance/7ab12d80-8a0d-4b00-9525-7ed3b39aca46 has 7,467mb of free memory. 2,048mb are required
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService waitForSufficientClusterResources
INFO: Instance arn:aws:ecs:us-east-1:318896727843:container-instance/7ab12d80-8a0d-4b00-9525-7ed3b39aca46 has 1,536 units of free cpu. 512 units are required
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Created Slave: slave-35123838f34f7
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService registerTemplate
INFO: Match on container definition: true
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService registerTemplate
INFO: Match on volumes: true
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService registerTemplate
INFO: Match on task role: true
Jul 03, 2018 3:02:55 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService registerTemplate
INFO: Match on execution role: false
Jul 03, 2018 3:02:56 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService registerTemplate
INFO: Created Task Definition: {TaskDefinitionArn: arn:aws:ecs:us-east-1:318896727843:task-definition/slave-slave:4,ContainerDefinitions: [{Name: slave-slave,Image: yadaraviteja/jenkins-agent:6.0,Cpu: 512,MemoryReservation: 2048,Links: [],PortMappings: [],Essential: true,EntryPoint: [],Command: [],Environment: [],MountPoints: [{SourceVolume: docker,ContainerPath: /var/run,ReadOnly: false}],VolumesFrom: [],Privileged: false,DnsServers: [],DnsSearchDomains: [],ExtraHosts: [],DockerSecurityOptions: [],Ulimits: [],}],Family: slave-slave,Revision: 4,Volumes: [{Name: docker,Host: {SourcePath: /var/run}}],Status: ACTIVE,RequiresAttributes: [{Name: com.amazonaws.ecs.capability.docker-remote-api.1.21,}],PlacementConstraints: [],Compatibilities: [EC2],RequiresCompatibilities: [],}
Jul 03, 2018 3:02:56 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Running task definition arn:aws:ecs:us-east-1:318896727843:task-definition/slave-slave:4 on slave slave-35123838f34f7
Jul 03, 2018 3:02:56 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: Slave slave-35123838f34f7 - Slave Task Started : arn:aws:ecs:us-east-1:318896727843:task/c0630e4d-1473-4853-b6b7-f00d60310e93
Jul 03, 2018 3:02:58 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #4 from /172.16.0.77:60956
Jul 03, 2018 3:03:04 PM com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback call
INFO: ECS Slave slave-35123838f34f7 (ecs task arn:aws:ecs:us-east-1:318896727843:task/c0630e4d-1473-4853-b6b7-f00d60310e93) connected
Jul 03, 2018 3:03:06 PM hudson.slaves.NodeProvisioner$2 run
INFO: ECS Slave slave provisioning successfully completed. We have now 3 computer(s)
Jul 03, 2018 3:03:06 PM hudson.model.Run execute
INFO: Sample-project #4 main build action completed: SUCCESS
Jul 03, 2018 3:03:06 PM com.cloudbees.jenkins.plugins.amazonecs.ECSComputer terminate
INFO: Attempting to terminate the node for computer: com.cloudbees.jenkins.plugins.amazonecs.ECSComputer@4417321d
Jul 03, 2018 3:03:06 PM com.cloudbees.jenkins.plugins.amazonecs.ECSComputer terminate
INFO: Terminating the node for computer: com.cloudbees.jenkins.plugins.amazonecs.ECSComputer@4417321d
Jul 03, 2018 3:03:06 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService deleteTask
INFO: Delete ECS Slave task: arn:aws:ecs:us-east-1:318896727843:task/c0630e4d-1473-4853-b6b7-f00d60310e93
Jul 03, 2018 3:03:06 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Executor #0 for slave-35123838f34f7 : executing Sample-project #4 for slave-35123838f34f7 terminated
java.nio.channels.ClosedChannelException
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
        at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
        at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
        at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
        at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
        at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
        at hudson.remoting.Channel.close(Channel.java:1446)
        at hudson.remoting.Channel.close(Channel.java:1399)
        at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746)
        at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:713)
        at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:88)
        at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:227)
        at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1545)
        at jenkins.model.Nodes$5.run(Nodes.java:246)
        at hudson.model.Queue._withLock(Queue.java:1375)
        at hudson.model.Queue.withLock(Queue.java:1252)
        at jenkins.model.Nodes.removeNode(Nodes.java:237)
        at jenkins.model.Jenkins.removeNode(Jenkins.java:2059)
        at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:70)
        at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.terminate(ECSComputer.java:92)
        at com.cloudbees.jenkins.plugins.amazonecs.ECSComputer.taskCompleted(ECSComputer.java:68)
        at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140)
        at hudson.model.Executor.finish1(Executor.java:476)
        at hudson.model.Executor.run(Executor.java:450)

Output of docker version:
EC2 instance Docker Version:

[root@ip 2]# docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

Jenkins Docker container docker version:

bash-4.2$ docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version:
 Go version:      go1.9.4
 Git commit:      94f4240/1.13.1
 Built:           Fri May 18 15:44:33 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version:
 Go version:      go1.7.5
 Git commit:      092cba3
 Built:           Wed Feb  8 06:38:28 2017
 OS/Arch:         linux/amd64
 Experimental:    false

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS
@ghost
Copy link

ghost commented Jul 3, 2018

I am also experiencing this issue when attempting to tag and push docker images from inside containers using this setup. When you issue the docker login from the host machine you get a successful login; however, over the docker.sock connection there seems to be some mechanic which forgets the login.

I also added a vmware harbor server to attempt the login and I receive success on the host but not from the container to host over the socket.

System versions are identical to what ravi1216 has posted above, built using the same Dockerfile.

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Jul 3, 2018

@skewled The login is stored on the client side (typically in a password database appropriate for the client platform) and passed to the daemon when a pull is attempted. In lieu of a supported password DB, the credentials are stored on disk in ~/.docker/config.json

@ravi1216 Is this a new repository? Do you have auto-creation of new repos disabled?

@ravi1216
Copy link
Author

ravi1216 commented Jul 3, 2018

I am using the old Repo

@ravi1216
Copy link
Author

ravi1216 commented Jul 3, 2018

yes it is storing the password ~/.docker/config.json location

@ravi1216
Copy link
Author

ravi1216 commented Jul 3, 2018

"Do you have auto-creation of new repos disabled?" can I know where can I find this option.

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Jul 7, 2018

can I know where can I find this option.

I'm mistaken, the feature I was thinking about is actually the default visibility for new repos (public or private).

@ravi1216
Copy link
Author

ravi1216 commented Jul 9, 2018

The default visibility for new repos "public".

@Bulletninja
Copy link

I'm having the same issue after successful login for gitlab's registry

@opadron opadron mentioned this issue Feb 7, 2019
Merged
@Freyert Freyert mentioned this issue Feb 26, 2020
Closed
@aalok-sathe
Copy link

This stackoverflow answer (https://stackoverflow.com/a/43862114/2434875) did the trick for me.
However, it is absolutely bizarre that it should do the trick, whereas the standard method of logging in should fail, so +1 to this bug report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/builder area/distribution version/1.13
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Bulletninja @cpuguy83 @GordonTheTurtle @aalok-sathe @ravi1216