Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[pawarrchetan]

Describe the bug
After using the AWS Secrets Backend in VAULT I am getting below error after using the correct credentials stored in VAULT.

[root@ip-100-66-12-90 tf]# vault read aws/creds/my-role

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

To Reproduce
Steps to reproduce the behavior:

1. vault write aws/config/root
   access_key=ABCDEFGHIJKL
   secret_key=AcVFGCbhjksluyCGD/JHGDPbps2X
   region=eu-central-1

-- Above entry is dummy keys.

vault write aws/roles/my-role
credential_type=iam_user
policy_document=-<<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"iam:DeleteAccessKey",
"iam:GetUserPolicy",
"iam:GetAccessKeyLastUsed",
"iam:UpdateAccessKey",
"iam:GetUser",
"iam:ListRolePolicies",
"iam:CreateAccessKey",
"iam:ListAccessKeys"
],
"Resource": "arn:aws:iam::80527613****:user/ark-key-rotate"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"iam:GenerateCredentialReport",
"iam:GetContextKeysForCustomPolicy",
"iam:ListPolicies",
"iam:ListUsers",
"iam:GetAccountAuthorizationDetails",
"iam:GetCredentialReport",
"iam:DeleteAccessKey",
"iam:GetUserPolicy",
"iam:GetAccessKeyLastUsed",
"iam:UpdateAccessKey",
"iam:GetUser",
"iam:ListRolePolicies",
"iam:CreateAccessKey",
"iam:ListAccessKeys"
],
"Resource": "*"
}
]
}
EOF
Success! Data written to: aws/roles/my-role

vault read aws/creds/my-role
Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Expected behavior
The output should contain the new temporary IAM credentials which can be used to login to AWS CLI.

Environment:

• Vault Server Version (retrieve with vault status):
  vault status
  Key Value

---

Seal Type shamir
Initialized true
Sealed false
Total Shares 3
Threshold 2
Version 0.11.4
Cluster Name vault-cluster-fb591cdc
Cluster ID a924d375-d667-0303-2d1d-4ecfaed1f4de
HA Enabled false

• Vault CLI Version (retrieve with vault version):
  vault version
  Vault v0.11.4 ('612120e76de651ef669c9af5e77b27a749b0dba3')

• Server Operating System/Architecture:
  Amazon Linux
  Linux ip-100-66-12-**.eu-central-1.compute.internal 4.14.42-61.37.amzn2.x86_64 Initial Website Import #1 SMP Mon May 21 23:43:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  Vault server configuration file(s):

backend "file" {
        path = "/opt/SP/apps/VAULT/data"
}

listener "tcp" {
        tls_disable = 1
}

Additional context
I would like to know what could be the problem here.
I have also added the credentials for the master user in the .aws/credentials file.
but the error still persists.

[pawarrchetan]

In the /var/log/messages file I can see below errors when I try to execute vault read aws/creds/my-role

Nov 30 12:33:04 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:34:22 ip-100-66-12-90 dhclient[3289]: XMT: Solicit on eth0, interval 111680ms.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] HealthCheck reporting agent health.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 ERROR [instanceID=i-0eb6e4936f0377681] [HealthCheck] error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] increasing error count by 1
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 ERROR [instanceID=i-0eb6e4936f0377681] [HealthCheck] error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated.
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] increasing error count by 1

[joelthompson]

Hi @pawarrchetan -- that error most likely means your box doesn't have network connectivity to be able to talk to the AWS IAM endpoints. Does your machine have outbound internet access (e.g., in a subnet with routes to either an IGW or NAT, a security group that permits outbound access to 0.0.0.0/0, no iptables rules on the machine blocking outbound internet access)?

[pawarrchetan]

@joelthompson
Thanks for your reply.

however my machine has access to the AWS API endpoints as I can easily use aws cli to communicate with the AWS resources.
Please see below :
$ nslookup www.google.com
Server: 100.66.12.2
Address: 100.66.12.2#53

Non-authoritative answer:
Name: www.google.com
Address: 172.217.22.4

I can list the S3 buckets in my account using aws cli.

aws s3api list-buckets

{
"Owner": {
"ID": "**************************a115287808afc4a02fb33f4ef4"
},
"Buckets": [
{
"CreationDate": "2018-09-10T07:59:04.000Z",
"Name": "2gj9i-g8s-access-logs"
},
{
"CreationDate": "2018-11-28T16:12:51.000Z",
"Name": "2mpte-g8s-access-logs"
},

Regards...
Chetan

[joelthompson]

The ListBucket command could be going through an S3 VPC Endpoint, while the IAM calls wouldn't. Can you use the awscli to make an IAM call, e.g., aws iam get-user?

[pawarrchetan]

@joelthompson
Yes I can do IAM calls using aws cli

aws iam get-user

{
"User": {
"UserName": "terraform-crp",
"Path": "/",
"CreateDate": "2018-11-08T11:20:01Z",
"UserId": "AIDAJJENJGUOS********",
"Arn": "arn:aws:iam::80527613****:user/terraform-crp"
}
}

[pawarrchetan]

@joelthompson

ANy update on the request ?

Regards...
Chetan

[gauthamatic]

I'm facing a similar issue but for Assume role credentials.
Any update on this?

[mdasilva]

I'm facing the same issue. Is this a confirmed bug?

[TomHowarth]

this is my code sanitized for here.

provider "vault" {
address = "${var.vault_addr}"
token = "${var.vault_token}"
}

resource "vault_aws_secret_backend" "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "us-east-1"

default_lease_ttl_seconds = "120"
max_lease_ttl_seconds = "240"
}

resource "vault_aws_secret_backend_role" "ec2-admin" {
backend = "${vault_aws_secret_backend.aws.path}"
name = "my-role"
credential_type = "assumed_role"

policy_document=<<EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "",
"Resource": ""
}
]
}
EOT
}

my vault is running and showing as initialized.

When I run the above code it is successful.

however when I run the following: vault read /aws/creds/my-role
I receive the following error

Error reading aws/creds/ec2-admin-role: Error making API request.

URL: GET http://127.0.0.1:8200/v1/aws/creds/my-role
Code: 400. Errors:

• did not supply a role_arn parameter and unable to determine one

[spangenberg]

We have two different issues here.

@TomHowarth your vault_aws_secret_backend is missing the allowed AWS roles to assume, please take a look here: https://www.terraform.io/docs/providers/vault/r/aws_secret_backend_role.html#role_arns

Regarding the first issue from @pawarrchetan, it looks networking related to me. If this still persists, I'd like to know more about the network configuration you're using, how the VPC is set up, which region you're running.

[spangenberg]

We haven't heard back regarding this issue in over 24 days. To try and keep our GitHub issues current, we'll be closing this issue in approximately seven days if we do not hear back regarding this issue. Please let us know if you can still reproduce this issue, and if there is any more information you could share, otherwise we'll be closing this issue.

[spangenberg]

Closing for now. Let us know if you have any more information!