bug: Error creating route when using multiple subnets in one vpc

[cgoIT]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

First of all I'm not a neither an AWS nor a terraform expert. So maybe I'm the problem and not localstack. ;)

I've created to vpc "foo" and "bar" where both vpcs have two subnets (a private and a public one). I want to setup vpc peering between those two vpcs. I'm able to create the peering connection. After that I would like to add the corresponding routes to the route tables where the subnets are associated to.

During the creation of the routes localstack complains about the fact, that the desired destination cidrs already exist. If I query the existing route tables I can see that this is not the case.

Expected Behavior

Routes should be created.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

version: '3.1'
services:
   localstack:
      image: localstack/localstack-pro:latest
      environment:
         - TZ=${TZ:-Europe/Berlin}
         - LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY:- }
         - REQUIRE_PRO=1
         - DEBUG=${DEBUG:-0}
      ports:
         - "127.0.0.1:4566-4583:4566-4583"
         - "127.0.0.1:53:53"
         - "127.0.0.1:443:443"
      volumes:
         - "/var/run/docker.sock:/var/run/docker.sock"

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

My terraform script:

###################### Peering Connection #####################

resource "aws_vpc_peering_connection" "foo_bar" {
  peer_vpc_id   = aws_vpc.bar.id
  vpc_id        = aws_vpc.foo.id
  auto_accept = true

  accepter {
    allow_remote_vpc_dns_resolution = true
  }

  requester {
    allow_remote_vpc_dns_resolution = true
  }
}



###################### Peering #####################

data "aws_route_table" "foo_public" {
  subnet_id = aws_subnet.foo_public.id
}

data "aws_route_table" "foo_private" {
  subnet_id = aws_subnet.foo_private.id
}

data "aws_route_table" "bar_public" {
  subnet_id = aws_subnet.bar_public.id
}

data "aws_route_table" "bar_private" {
  subnet_id = aws_subnet.bar_private.id
}


resource "aws_route" "foo_public_to_bar" {
  route_table_id            = data.aws_route_table.foo_public.id
  destination_cidr_block    = aws_vpc.bar.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.foo_bar.id
  depends_on                = [aws_vpc_peering_connection.foo_bar]
}

resource "aws_route" "foo_private_to_bar" {
  route_table_id            = data.aws_route_table.foo_private.id
  destination_cidr_block    = aws_vpc.bar.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.foo_bar.id
  depends_on                = [aws_vpc_peering_connection.foo_bar]
}

resource "aws_route" "bar_public_to_foo" {
  route_table_id            = data.aws_route_table.bar_public.id
  destination_cidr_block    = aws_vpc.foo.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.foo_bar.id
  depends_on                = [aws_vpc_peering_connection.foo_bar]
}

resource "aws_route" "bar_private_to_foo" {
  route_table_id            = data.aws_route_table.bar_private.id
  destination_cidr_block    = aws_vpc.foo.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.foo_bar.id
  depends_on                = [aws_vpc_peering_connection.foo_bar]
}


###################### VPCs #####################

resource "aws_vpc" "foo" {
  cidr_block = "10.1.0.0/16"
}

resource "aws_vpc" "bar" {
  cidr_block = "10.2.0.0/16"
}


###################### Subnets #####################

resource "aws_subnet" "foo_public" {
  vpc_id            = aws_vpc.foo.id
  cidr_block        = "10.1.1.0/24"
}

resource "aws_subnet" "foo_private" {
  vpc_id            = aws_vpc.foo.id
  cidr_block        = "10.1.2.0/24"
}

resource "aws_subnet" "bar_public" {
  vpc_id            = aws_vpc.bar.id
  cidr_block        = "10.2.1.0/24"
}

resource "aws_subnet" "bar_private" {
  vpc_id            = aws_vpc.bar.id
  cidr_block        = "10.2.2.0/24"
}


#################### Internet Gateways #################

resource "aws_internet_gateway" "foo" {
  vpc_id = aws_vpc.foo.id
}

resource "aws_internet_gateway" "bar" {
  vpc_id = aws_vpc.bar.id
}


################### Elastic IPs ###########
resource "aws_eip" "foo" {
  vpc = true
}

resource "aws_eip" "bar" {
  vpc = true
}


################### Nat gateways ###########################
resource "aws_nat_gateway" "foo" {
  allocation_id = aws_eip.foo.id
  subnet_id     = aws_subnet.foo_public.id
  depends_on    = [aws_internet_gateway.foo]
}

resource "aws_nat_gateway" "bar" {
  allocation_id = aws_eip.bar.id
  subnet_id     = aws_subnet.bar_public.id
  depends_on    = [aws_internet_gateway.bar]
}


################### Route tables ###########################
resource "aws_route_table" "foo_public" {
  vpc_id = aws_vpc.foo.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.foo.id
  }

  lifecycle {
    ignore_changes = [route]
  }
}

resource "aws_route_table_association" "foo_public_association" {
  subnet_id      = aws_subnet.foo_public.id
  route_table_id = aws_route_table.foo_public.id
}

resource "aws_route_table" "foo_private" {
  vpc_id = aws_vpc.foo.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.foo.id
  }

  lifecycle {
    ignore_changes = [route]
  }
}

resource "aws_route_table_association" "foo_private_association" {
  subnet_id      = aws_subnet.foo_private.id
  route_table_id = aws_route_table.foo_private.id
}

resource "aws_route_table" "bar_public" {
  vpc_id = aws_vpc.bar.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.bar.id
  }

  lifecycle {
    ignore_changes = [route]
  }
}

resource "aws_route_table_association" "bar_public_association" {
  subnet_id      = aws_subnet.bar_public.id
  route_table_id = aws_route_table.bar_public.id
}

resource "aws_route_table" "bar_private" {
  vpc_id = aws_vpc.bar.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.bar.id
  }

  lifecycle {
    ignore_changes = [route]
  }
}

resource "aws_route_table_association" "bar_private_association" {
  subnet_id      = aws_subnet.bar_private.id
  route_table_id = aws_route_table.bar_private.id
}

Error messages:

Error: creating Route in Route Table (rtb-8f86de50) with destination (10.2.0.0/16): RouteAlreadyExists: The route identified by 10.2.0.0/16 already exists
│       status code: 400, request id: QQVFX2CHN4S7FUVC9S47D12476SS9NYSGC7A728LPYM2E80LCRW5
│ 
│   with aws_route.foo_public_to_bar,
│   on main.tf line 38, in resource "aws_route" "foo_public_to_bar":
│   38: resource "aws_route" "foo_public_to_bar" {
│ 
╵
╷
│ Error: creating Route in Route Table (rtb-1ead2371) with destination (10.2.0.0/16): RouteAlreadyExists: The route identified by 10.2.0.0/16 already exists
│       status code: 400, request id: OIJH1KXVDRZOHWYK79C3EUFGY20P4W4F7N5E9PNQ0X0EQAYI7BD0
│ 
│   with aws_route.foo_private_to_bar,
│   on main.tf line 45, in resource "aws_route" "foo_private_to_bar":
│   45: resource "aws_route" "foo_private_to_bar" {
│ 
╵
╷
│ Error: creating Route in Route Table (rtb-1b78a2e6) with destination (10.1.0.0/16): RouteAlreadyExists: The route identified by 10.1.0.0/16 already exists
│       status code: 400, request id: M1BGCCNKDR0EY3VSIM60CLEZ6RTXYXTJKNUKTN7F61UIVQSF0W5I
│ 
│   with aws_route.bar_public_to_foo,
│   on main.tf line 52, in resource "aws_route" "bar_public_to_foo":
│   52: resource "aws_route" "bar_public_to_foo" {
│ 
╵
╷
│ Error: creating Route in Route Table (rtb-75ffb3bf) with destination (10.1.0.0/16): RouteAlreadyExists: The route identified by 10.1.0.0/16 already exists
│       status code: 400, request id: JJVIAKXMYNEQ3IF2P5IVTVORBEYIJT5XHSL2BYAQX7C3UV56LG67
│ 
│   with aws_route.bar_private_to_foo,
│   on main.tf line 59, in resource "aws_route" "bar_private_to_foo":
│   59: resource "aws_route" "bar_private_to_foo" {

Output of "awslocal ec2 describe-route-tables --filter Name=route-table-id,Values=rtb-8f86de50"

{
  "RouteTables": [
    {
      "Associations": [
        {
          "Main": false,
          "RouteTableAssociationId": "rtbassoc-7be6934d",
          "RouteTableId": "rtb-8f86de50",
          "SubnetId": "subnet-3ba9d3c0",
          "AssociationState": {
            "State": "associated"
          }
        }
      ],
      "RouteTableId": "rtb-8f86de50",
      "Routes": [
        {
          "DestinationCidrBlock": "10.1.0.0/16",
          "GatewayId": "local",
          "Origin": "CreateRouteTable",
          "State": "active"
        },
        {
          "DestinationCidrBlock": "0.0.0.0/0",
          "GatewayId": "igw-e1f33d15",
          "Origin": "CreateRoute",
          "State": "active"
        }
      ],
      "Tags": [],
      "VpcId": "vpc-f8d94257",
      "OwnerId": "000000000000"
    }
  ]
}

Environment

- OS: MacOS Ventura 13.2.1
- LocalStack: 1.4.1

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[cgoIT]

After some debugging I think that the source of the problem could be in the moto library. I've opened a ticket there too: getmoto/moto#6096

[cgoIT]

The bug in moto was fixed today (getmoto/moto#6106). I don't know if something has to be done on the localstack side, so this is just an information here.