ipv6 address used as extip results in 127.0.0.1 being used

[LeoHLee]

System information

Geth version: 1.13.14
CL client & version: e.g. nimbus@v24.3.0
OS & Version: Linux

Expected behaviour

Sync mainnet state and chaindata after startup.

Actual behaviour

INFO [04-09|11:51:38.790] Logging configured                       rotate=true format=terminal location=/home/leo_h/eth/geth.log
INFO [04-09|11:51:38.801] Starting Geth on Ethereum mainnet...
INFO [04-09|11:51:38.802] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [04-09|11:51:38.803] Maximum peer count                       ETH=50 total=50
INFO [04-09|11:51:38.805] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [04-09|11:51:38.811] Set global gas cap                       cap=50,000,000
INFO [04-09|11:51:38.811] Initializing the KZG library             backend=gokzg
INFO [04-09|11:51:38.867] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [04-09|11:51:38.867] Using pebble as the backing database
INFO [04-09|11:51:38.867] Allocated cache and file handles         database=/home/leo_h/eth/geth-data/geth/chaindata cache=2.00GiB handles=524,288
INFO [04-09|11:51:38.944] Opened ancient database                  database=/home/leo_h/eth/geth-data/geth/chaindata/ancient/chain readonly=false
INFO [04-09|11:51:38.944] State scheme set by user                 scheme=path
INFO [04-09|11:51:38.946] Initialising Ethereum protocol           network=1 dbversion=8
WARN [04-09|11:51:38.946] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
INFO [04-09|11:51:38.967] Opened ancient database                  database=/home/leo_h/eth/geth-data/geth/chaindata/ancient/state readonly=false
WARN [04-09|11:51:38.967] Path-based state scheme is an experimental feature
INFO [04-09|11:51:38.969]
INFO [04-09|11:51:38.969] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [04-09|11:51:38.969] Chain ID:  1 (mainnet)
INFO [04-09|11:51:38.969] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [04-09|11:51:38.969]
INFO [04-09|11:51:38.969] Pre-Merge hard forks (block based):
INFO [04-09|11:51:38.969]  - Homestead:                   #1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [04-09|11:51:38.969]  - DAO Fork:                    #1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [04-09|11:51:38.969]  - Tangerine Whistle (EIP 150): #2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [04-09|11:51:38.969]  - Spurious Dragon/1 (EIP 155): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [04-09|11:51:38.969]  - Spurious Dragon/2 (EIP 158): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [04-09|11:51:38.969]  - Byzantium:                   #4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [04-09|11:51:38.969]  - Constantinople:              #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [04-09|11:51:38.969]  - Petersburg:                  #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [04-09|11:51:38.969]  - Istanbul:                    #9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [04-09|11:51:38.969]  - Muir Glacier:                #9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [04-09|11:51:38.969]  - Berlin:                      #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [04-09|11:51:38.969]  - London:                      #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [04-09|11:51:38.969]  - Arrow Glacier:               #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [04-09|11:51:38.969]  - Gray Glacier:                #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [04-09|11:51:38.969]
INFO [04-09|11:51:38.969] Merge configured:
INFO [04-09|11:51:38.969]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [04-09|11:51:38.969]  - Network known to be merged: true
INFO [04-09|11:51:38.969]  - Total terminal difficulty:  58750000000000000000000
INFO [04-09|11:51:38.969]
INFO [04-09|11:51:38.969] Post-Merge hard forks (timestamp based):
INFO [04-09|11:51:38.969]  - Shanghai:                    @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [04-09|11:51:38.969]  - Cancun:                      @1710338135 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [04-09|11:51:38.969]
INFO [04-09|11:51:38.969] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [04-09|11:51:38.969]
INFO [04-09|11:51:38.969] Loaded most recent local block           number=0 hash=d4e567..cb8fa3 td=17,179,869,184 age=55y3w1d
INFO [04-09|11:51:38.970] Initialized transaction indexer          range="last 2350000 blocks"
INFO [04-09|11:51:38.971] Loaded local transaction journal         transactions=0 dropped=0
INFO [04-09|11:51:39.062] Enabled snap sync                        head=0 hash=d4e567..cb8fa3
INFO [04-09|11:51:39.063] Chain post-merge, sync via beacon client
INFO [04-09|11:51:39.063] Gasprice oracle is ignoring threshold set threshold=2
WARN [04-09|11:51:39.068] Engine API enabled                       protocol=eth
INFO [04-09|11:51:39.068] Starting peer-to-peer node               instance=Geth/v1.13.14-stable-2bd6bd01/linux-amd64/go1.21.6
INFO [04-09|11:51:39.088] New local node record                    seq=1,712,633,593,960 id=da0a4563386118c2 ip=127.0.0.1 udp=30303 tcp=30303
INFO [04-09|11:51:39.089] Started P2P networking                   self=enode://e44055b1a9dc317006443ea3ad938d039434a995b47c2f200b4c060c1e35aa708217330030b1674a95e1c88547e7dd47d71785ca90d6f18f5cbbda4fbb88611a@127.0.0.1:30303
INFO [04-09|11:51:39.090] IPC endpoint opened                      url=/home/leo_h/eth/geth-data/geth.ipc
INFO [04-09|11:51:39.091] Loaded JWT secret file                   path=/home/leo_h/eth/jwtsecret.hex crc32=0x5050dcd5
INFO [04-09|11:51:39.093] HTTP server started                      endpoint=127.0.0.1:8545 auth=false prefix= cors= vhosts=localhost
INFO [04-09|11:51:39.093] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [04-09|11:51:39.094] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=localhost
INFO [04-09|11:51:49.605] Looking for peers                        peercount=0 tried=1 static=0
WARN [04-09|11:52:14.069] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
INFO [04-09|11:52:14.090] Looking for peers                        peercount=0 tried=3 static=0
WARN [04-09|11:52:36.702] Ignoring payload while snap syncing      number=19,615,500 hash=2cc775..e0b643 reason="forced head needed for startup"
INFO [04-09|11:52:36.909] Forkchoice requested sync to new head    number=19,615,500 hash=2cc775..e0b643 finalized=unknown
INFO [04-09|11:53:37.082] Forkchoice requested sync to new head    number=19,615,501 hash=5761bf..5f95a8 finalized=unknown
INFO [04-09|11:53:37.324] Forkchoice requested sync to new head    number=19,615,502 hash=e0be89..a5a842 finalized=unknown
INFO [04-09|11:53:37.602] Forkchoice requested sync to new head    number=19,615,503 hash=4dfad7..40d86c finalized=unknown
INFO [04-09|11:53:38.784] Forkchoice requested sync to new head    number=19,615,504 hash=775f64..505b63 finalized=unknown
INFO [04-09|11:53:39.167] Forkchoice requested sync to new head    number=19,615,505 hash=5f7cfb..5d26c6 finalized=unknown
INFO [04-09|11:53:39.425] Forkchoice requested sync to new head    number=19,615,506 hash=354bd3..dd86e6 finalized=unknown
INFO [04-09|11:53:39.694] Forkchoice requested sync to new head    number=19,615,507 hash=d8dd88..1e5a5b finalized=unknown
INFO [04-09|11:53:39.955] Forkchoice requested sync to new head    number=19,615,508 hash=e4a8d6..133af3 finalized=unknown
INFO [04-09|11:53:40.238] Forkchoice requested sync to new head    number=19,615,509 hash=fa9ecc..45671a finalized=unknown
INFO [04-09|11:53:40.604] Forkchoice requested sync to new head    number=19,615,510 hash=6cbdb0..4889b3 finalized=unknown
INFO [04-09|11:53:40.904] Forkchoice requested sync to new head    number=19,615,511 hash=f4e9d4..928c45 finalized=unknown
INFO [04-09|11:53:41.163] Forkchoice requested sync to new head    number=19,615,512 hash=e5c39f..8fe807 finalized=unknown
INFO [04-09|11:53:41.423] Forkchoice requested sync to new head    number=19,615,513 hash=7c8d0f..445ff1 finalized=unknown
INFO [04-09|11:53:41.698] Forkchoice requested sync to new head    number=19,615,514 hash=5ae518..535c58 finalized=unknown
INFO [04-09|11:53:41.966] Forkchoice requested sync to new head    number=19,615,515 hash=899ec3..8389f8 finalized=unknown
INFO [04-09|11:53:42.230] Forkchoice requested sync to new head    number=19,615,516 hash=900940..b62ed8 finalized=unknown
INFO [04-09|11:53:42.506] Forkchoice requested sync to new head    number=19,615,517 hash=897d10..7a4da4 finalized=unknown
INFO [04-09|11:53:42.712] Forkchoice requested sync to new head    number=19,615,518 hash=883ad8..f38a5a finalized=unknown
INFO [04-09|11:53:42.945] Forkchoice requested sync to new head    number=19,615,519 hash=1eb1da..181578 finalized=unknown
INFO [04-09|11:53:43.175] Forkchoice requested sync to new head    number=19,615,520 hash=0dca13..41cf5a finalized=unknown
INFO [04-09|11:53:43.395] Forkchoice requested sync to new head    number=19,615,521 hash=61a34a..2e9ab4 finalized=unknown
INFO [04-09|11:53:43.666] Forkchoice requested sync to new head    number=19,615,522 hash=68d7e3..a5cc4b finalized=unknown
INFO [04-09|11:53:43.967] Forkchoice requested sync to new head    number=19,615,523 hash=56bc35..fdd1e4 finalized=unknown
INFO [04-09|11:53:44.193] Forkchoice requested sync to new head    number=19,615,524 hash=955670..87a068 finalized=unknown

Not syncing state and chain data, even after nimbus has synced and backfilled itself.

One line in the log says "forced head needed for startup".

Steps to reproduce the behaviour

Sync a nimbus CL using checkpoint with --backfill=false, then launch both geth and nimbus.

[namiloh]

Please provide the command line options used for starting geth

[LeoHLee]

Please provide the command line options used for starting geth

geth --datadir /home/leo_h/eth/geth-data --state.scheme=path --log.file /home/leo_h/eth/geth.log --log.rotate --authrpc.addr localhost --authrpc.port 8551 --authrpc.vhosts localhost --authrpc.jwtsecret /home/leo_h/eth/jwtsecret.hex --http --http.api admin,eth,net --v5disc --nat extip:$MY_IPv6_ADDR --bootnodes enode://bcf5de28545207b27b1bd62553701ed4b3b4662e5f9d44f7f6568a3009fa5702a9667d4775a172f5a59ea4962446ab7f0bcd7e852ea95c0b92f100bc99e26b8d@198.37.108.123:30303,enode://b8bd893b45a6c5a7f2dea872bf194c5b2596a9d9bf7b2d3c19912718e061c490382733d1ee5a6d61cba12ad3715a579f20904bb6f78104b454910c5a59a0adf1@142.132.251.135:30303,enode://2e62394968594bec2d04bff3fe847929cd8b5306f59c95243cb9775769fda1a21c0585ed203c10b7aa2dce229c22cb05a1e288d21f78544a72211cac79e78f21@107.149.163.66:30303,enode://f673a314639da5889042721350e66110b3a0f1ba757c60655db67b9b64efd17649e23fef6531999e18b6a6c4c8068dec775b6b7f1a88b5026c594f55e43ac2f0@100.26.35.54:30303

I just tried removing --nat and --bootnodes, and it can sync now. Title of this issue has been changed

[holiman]

The --bootnodes should not matter, however, --nat is probably the culprit. You are explicitly giving a nat address: telling geth "hey this is my external IP, don't bother trying to figure it out by yourself".

We can't see what it resolves to, since $MY_IPv6_ADDR is a bash expansion. But I guess you try to set an ipv6 address. I assume that geth doesn't support that, and also that geth doesn't bother figuring out the ipv4 address in this case, and it leaves you with an ext ip of 127.0.0.1: nobody will be able to send data to you. @fjl makes sense?

[holiman]

Yes, it seems something doesn't quite work when using an ext ipv6 address:

[user@work go-ethereum]$ go run ./cmd/geth --nodiscover --maxpeers 0 --nat extip:2001:db8:0:1:1:1:1:1 2>&1 | grep "New local"
INFO [04-09|09:13:35.775] New local node record                    seq=1,664,353,517,476 id=1764c0f9d97dff96 ip=127.0.0.1 udp=0 tcp=30303

[holiman]

The node "kind of" has support for ipv6, but it prefers ipv4.

// IP returns the IP address of the node. This prefers IPv4 addresses.
func (n *Node) IP() net.IP {
	var (
		ip4 enr.IPv4
		ip6 enr.IPv6
	)
	if n.Load(&ip4) == nil {
		return net.IP(ip4)
	}
	if n.Load(&ip6) == nil {
		return net.IP(ip6)
	}
	return nil
}

However, we also always set a fallback ipv4 ip to 127.0.0.1:

	srv.localnode.SetFallbackIP(net.IP{127, 0, 0, 1})

So, an explicit ipv6 is set, preventing auto-discovery. And 127.0.0.1 set as fallback, in this case it takes precedence over the one you set explicitly.