Failure to create correct firewall rules under Debian 12 ("Bookworm")

[this-user]

Description

Docker does create the DOCKER-USER chain along with a jump rule at the top of the FORWARD chain to the aforementioned chain. This does work as expected under Debian 11 and under Debian 12 as long as the DOCKER-USER chain does not yet exist when Docker is being started.

However, if the DOCKER-USER chain already does exist, e.g. because it has been manually created, the Docker services produces an error message during startup, indicating that the chain already exists.

level=warning msg="Failed to create DOCKER-USER IPV4 chain" error="iptables failed: iptables -t filter -N DOCKER-USER: iptables: Chain already exists.\n (exit status 1)"

More importantly, this leads to the knock-on effect that no jump rule at the top of the FORWARD chain is created, leading to a situation where the DOCKER-USER chain is effectively being ignored.

Using the same Docker version and configuration, this issue does not occur under Debian 11 even if the DOCKER-USER chain already exists, it only occurs under Debian 12 if the chain exists already.

---

The most likely explanation seems to be different behaviour by at least one of the packages on the OS side to which Docker reacts slightly differently, and does not create the jump rule.

Reproduce

1. iptables -t filter -N DOCKER-USER
2. systemctl start docker.service
3. nft list ruleset

-> jump rule in FORWARD chain is missing

Expected behavior

Jump rule to DOCKER-USER at the top of the FORWARD chain exists.

docker version

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:35 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:35 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 3
  Paused: 0
  Stopped: 2
 Images: 8
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-10-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 10
 Total Memory: 31.34GiB
 Name: <MY DOMAIN>
 ID: <MY ID>
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Debian 12 versions:

• debian_version: 12.1
• iptables v1.8.9 (nf_tables)
• nftables v1.0.6 (Lester Gooch # 5)

Debian 11 versions:

• debian_version: 11.7
• iptables v1.8.7 (nf_tables)
• nftables v0.9.8 (E.D.S.)

[akerouanton]

Hi @this-user,

Since you're mentioning iptables-nft in the Additional Info section, could you try to switch to iptables-legacy with the following commands:

# Select the legacy option each time
$ update-alternatives --config iptables
$ update-alternatives --config ip6tables
# Then:
$ sudo systemctl restart docker.service

If that doesn't work, could you please enable debug log (see https://docs.docker.com/config/daemon/logs/#enable-debugging), restart your daemon once more (the kill -SIGHUP mentioned in the docs won't help here) and then copy/paste or upload a text file of what's logged by the daemon at startup?

[this-user]

I have tried switching the iptables backend. This does solve the issue of Docker producing an error during startup, even if a DOCKER-USER chain exists in the iptables backend. Unfortunately, this doesn't really help with doing what I am actually trying to do with that chain.

What's more, Debian 11 has always been using the iptables-nft backend instead of the legacy one, and that has never been a problem. So the issue has to be that either the Docker build for Debian 12 is slightly different, even though it is the exact same version, or there are differences in the OS packages. But the difference in version between Debian 11 and 12 of the iptables and nftables packages are not huge.

I have created the log, it is attached below. The key aspect there is the error message regarding the DOCKER-USER chain that already exists, which does not happen on Debian 11.

level=warning msg="Failed to create DOCKER-USER IPV4 chain" error="iptables failed: iptables -t filter -N DOCKER-USER: iptables: Chain already exists.\n (exit status 1)"

docker.log

---

EDIT: There is also another issue early on where the daemon cannot determine whether iptables supports xlock. This seems to be an issue with the 1.8.9 version of that packages. Maybe this is related, and is what trips up Docker.

https://bugzilla.netfilter.org/show_bug.cgi?id=1632

[this-user]

I think I understand most of the issue. It's not the existence of the DOCKER-USER chain, it's the fact that I have a meta rule in that chain already. That rule causes an error with the iptables command in version 1.8.9 (and probably 1.8.8 too), 1.8.7 (which is what Deb 11 uses) is fine.

This error then trips up the startup of the Docker daemon, causing the first issue with being unable to determine the xlock capabilities. This, in turn, seems to change how Docker handles the rest of the FW initialisation, which then causes an error when it tries to create the already existing DOCKER-USER chain. What I do not fully understand is how this second issue connects to the first one, as the iptables -t filter N DOCKER-USER command does not produce an error even if the meta rule already exists in that chain. So it has to be something that is caused by the first issue.

Below is a log of starting the Docker daemon on Debian 12 with an existing but empty DOCKER-USER chain, which works as expected.

docker_2.log

Unfortunately, there seems to be no good way of fixing this, as Debian 12 only supports iptables 1.8.9, and there is no newer upstream version yet.

[fclaerho]

I'm confirming hitting the exact same issue.
Took a while to understand what was happening and finally hit this bug report.
The problem occurs only a machine with Debian 12. Same log entry:

`level=warning msg="Failed to create DOCKER-USER IPV4 chain" error="iptables failed: iptables -t filter -N DOCKER-USER: iptables: Chain already exists.\n (exit status 1)`

for the same reason: I have an customized DOCKER-USER chain configured prior to having Docker generate its rules.
The outcome is as described above, we miss the jump DOCKER-USER in the FORWARD chain, leading to containers connected to the bridge network not being able to reach anything outside. Containers connected to the host network are working as expected.

I'll try to either drop my customised content or insert manually the missing command somewhere:

iptables -I FORWARD -j DOCKER-USER

[this-user]

It's a problem with the iptables package that was introduced in version 1.8.9. I can't remember the specific commit, but it had something to do with meta rules, I think. Anyway, the best solution for the time being on Debian 12 seems to be to build the 1.8.8 version of the packages iptables, libip4tc2, libip6tc2, and libxtables12 yourself, install them, and block updates. Hopefully, upstream will eventually get around to fixing this. But even then, it might take Debian some time to update the package, too.

EDIT: They do have a Bugzilla issue open for this, but unfortunately, this is one of those annoying projects that make it really difficult to reach them.

[thaJeztah]

To save others from following the link; it's closed as WONTFIX;

Compatibility between iptables-nft and nftables can't be "fixed", many
expressions in nftables rules can't be translated into iptables syntax as it
simply lacks the necessary capabilities.

The specific problem illustrated here (setting packet mark) is fixed by commit
7304f1982d619 ("nft-ruleparse: parse meta mark set as MARK target"), enabling
iptables-nft to correctly parse the meta mark statement.

Improving the iptables-nft parser to understand more native expressions is a
task actively being worked on, but mixing iptables-nft and nftables will always
remain problematic and a good way to shoot one's own foot!

[akerouanton]

The linked Bugzilla issue mentions this bug is triggered by mixing both nftables and iptables-nft rules. I believe you need to do some deliberate action to end up mixing both, and given that there're already a bunch of resources warning that anyone trying to do that would end up in troubled waters, I consider there's not much we can do here. We might consider adding a warning into our docs if we find more users are affected by this issue or alike.

So let me close this one, but feel free to continue the discussion if you find something interesting or if I said something wrong.

[fclaerho]

I've been trying to catch up on fixing something I didn't setup -- so could you please clarify the "expected" way of doing things here?
I'm not a fan of either solution but it seems Netfilter has decided that nftables is supposed to be way forward and deprecates iptables -- and as far as I understand so far, the only way of having Docker collaborates is one of the iptables/nft adapters.
Is the conclusion that we have to choose between either Docker, or nftables here?

[this-user]

The real problem is that Docker does not currently support nftables directly and instead relies on the nftables-iptables compatibility layer on systems that are using nftables when it is managing the firewall rules. This is what necessitates the workaround with the meta mark rule in the first place. Unfortunately, that workaround is now broken when using version 1.8.9 of the iptables packages. The only other option except for using version 1.8.8 would be to disable Docker's handling of FW rules and manage them manually.

But the real solution would be for Docker to implement direct support of nftables. Then we would not need the workaround in the first place, and we would not run into this bug. Given that nftables is effectively the designated successor of iptables, it is to be expected that more distributions will make the switch eventually which is a good reason why Docker should be able to work with it.

[akerouanton]

Is the conclusion that we have to choose between either Docker, or nftables here?

More or less, yes. Or rather, you have to choose between iptables-nft and iptables-legacy, and nft is ruled out.

To be more precise, Docker is only compatible with iptables-legacy and iptables-nft. If you create some rules with nft, those might not be parsed correctly by iptables-nft and that leads to some variant of the original error message reported here. As noted in Phil Sutter's answer, mixing both iptables-nft and nft is highly discouraged. So you effectively have no other choice than using either iptables-nft or iptables-legacy system-wide when Docker is installed.

Improving the iptables-nft parser to understand more native expressions is a
task actively being worked on, but mixing iptables-nft and nftables will always
remain problematic and a good way to shoot one's own foot!

We're aware we need to tackle this issue (see #26824), but we have some important overhauling to go through first as the way iptables rules are managed right now is quite messy. That's an unfortunate situation, but we can't do better right now.

This is what necessitates the workaround with the meta mark rule in the first place.

@this-user Can you expand on that? I don't get why this is needed.

Looking at https://wiki.debian.org/nftables, I see:

the iptables utility may not be installed in a system by default.

I guess the corollary is nft might be installed by default on those systems. Looking at our installation guide for Debian, I don't see any warning about that. That makes me change my mind, and I now think it's worth mentioning it in our docs. Let me reopen this issue and friendly ping @dvdksn.

[this-user]

@akerouanton

Can you expand on that? I don't get why this is needed.

If you install Docker per its documentation on a system with nftables like the most recent versions of Debian, the containers simply will not be able to create any outgoing connections, because their traffic gets stuck in the nftables inet filter forward chain, unless you set its policy to ACCEPT, which you probably shouldn't. The reason for that is that Docker cannot touch that chain as it is using the iptables legacy mode and can therefore only modify the iptables FORWARD chain.

Thus, you can either disable Docker's management of FW rules altogether and do it manually, or you can do a workaround like adding one rule to the DOCKER-USER chain to set a mark that identifies Docker traffic, and then add a second rule in the nftables forward chain to accept traffic with that flag. Then it works as expected.

In any case, the bottom line is that traffic forwarding from containers does not currently work out of the box if the system is using nftables with anything but an accept policy for its forward chain if Docker is managing the firewall rules. That is arguably something that ought to be fixed on the Docker side.

[fclaerho]

Is the conclusion that we have to choose between either Docker, or nftables here?

More or less, yes. Or rather, you have to choose between iptables-nft and iptables-legacy, and nft is ruled out.

Alright, thanks for taking the time to reply.

We're aware we need to tackle this issue (see #26824), but we have some important overhauling to go through first as the way iptables rules are managed right now is quite messy. That's an unfortunate situation, but we can't do better right now.

No problem, I know this is not trivial; the summary is that we are all in a bad spot but it's not really the fault of anyone, just the general entropy of the universe :-)
Anyway, I got a working workaround, and other options are mentioned in this thread so people can refer to that;

Lastly, I can understand you don't want to keep this ticket lingering around, I'd imagine it could possibly be marked as dup of a more general ticket related to the Docker/Nft upcoming support? something like that.

[christianbur]

I also have the problem that my own nftables rules (/etc/nftables.conf) are not loaded correctly on restart.

# journalctl | grep "Failed to create DOCKER-USER"
Dec 27 01:05:46 leon dockerd[837]: time="2023-12-27T01:05:46.780516480+01:00" level=warning msg="Failed to create DOCKER-USER IPV4 chain" error="iptables failed: iptables -t filter -N DOCKER-USER: iptables: Chain already exists.\n (exit status 1)"

My solution is to create my own chain FORWARD-CB.
Although I create the DOCKER-USER chain and fill it with my own rule, docker does not interfere with it.

add chain ip  filter DOCKER-USER
add chain ip6 filter DOCKER-USER
insert rule ip  filter DOCKER-USER counter jump FORWARD-CB
insert rule ip6 filter DOCKER-USER counter jump FORWARD-CB

The flow is as follows:

chain FORWARD -> jump DOCKER-USER ->
chain DOCKER-USER -> jump FORWARD-CB ->
chain FORWARD-CB -> return (DOCKER-USER) -> return (FORWARD)

my /etc/nftabes.conf

#!/usr/sbin/nft -f

define IF-INET = enp3s0

add table ip  filter
add table ip6 filter
add chain ip  filter INPUT   { type filter hook input   priority 0; policy accept; }
add chain ip6 filter INPUT   { type filter hook input   priority 0; policy accept; }
add chain ip  filter OUTPUT  { type filter hook output  priority 0; policy accept; }
add chain ip6 filter OUTPUT  { type filter hook output  priority 0; policy accept; }
add chain ip  filter FORWARD { type filter hook forward priority 0; policy accept; }
add chain ip6 filter FORWARD { type filter hook forward priority 0; policy accept; }
add chain ip  filter DOCKER-USER
add chain ip6 filter DOCKER-USER
add chain ip  filter FORWARD-CB
add chain ip6 filter FORWARD-CB

flush chain ip  filter INPUT
flush chain ip6 filter INPUT
flush chain ip  filter OUTPUT
flush chain ip6 filter OUTPUT
flush chain ip  filter DOCKER-USER
flush chain ip6 filter DOCKER-USER
flush chain ip  filter FORWARD-CB
flush chain ip6 filter FORWARD-CB


insert rule ip  filter DOCKER-USER counter jump FORWARD-CB
insert rule ip6 filter DOCKER-USER counter jump FORWARD-CB

# these rules are set automatically by docker
#insert rule ip  filter DOCKER-USER counter return
#insert rule ip6 filter DOCKER-USER counter return

#################################################################################################################################
# INPUT
#################################################################################################################################

# already established connections
add rule ip  filter INPUT iifname $IF-INET ct state related,established counter accept
add rule ip6 filter INPUT iifname $IF-INET ct state related,established counter accept

# icmp
add rule ip  filter INPUT iifname $IF-INET icmp   type echo-request counter accept
add rule ip6 filter INPUT iifname $IF-INET icmpv6 type echo-request counter accept
add rule ip6 filter INPUT iifname $IF-INET icmpv6 type { nd-neighbor-advert, nd-neighbor-solicit, nd-router-advert} counter accept

# broadcast,multicast
add rule ip  filter INPUT iifname $IF-INET pkttype { broadcast,multicast} counter accept
add rule ip6 filter INPUT iifname $IF-INET pkttype { broadcast,multicast} counter accept

# ssh
add rule ip  filter INPUT iifname $IF-INET ct state new tcp dport 22 counter accept comment "ssh"
add rule ip6 filter INPUT iifname $IF-INET ct state new tcp dport 22 counter accept comment "ssh"

# logging
add rule ip  filter INPUT iifname $IF-INET counter log prefix "nft-drop-INPUT: " flags tcp options flags ip options
add rule ip6 filter INPUT iifname $IF-INET counter log prefix "nft-drop-INPUT: " flags tcp options flags ip options

# drop all
add rule ip  filter INPUT iifname $IF-INET counter drop comment "drop-all"
add rule ip6 filter INPUT iifname $IF-INET counter drop comment "drop-all"


#################################################################################################################################
# FORWARD-CB (in DOCKER-USER chain)
#################################################################################################################################

# already established connections
add rule ip  filter FORWARD-CB iifname $IF-INET ct state related,established counter accept
add rule ip6 filter FORWARD-CB iifname $IF-INET ct state related,established counter accept

# proxy-web (docker container)
add rule ip  filter FORWARD-CB iifname $IF-INET ct state new meta l4proto { tcp, udp } th dport { 80, 443 } counter accept comment "proxy-web"
add rule ip6 filter FORWARD-CB iifname $IF-INET ct state new meta l4proto { tcp, udp } th dport { 80, 443 } counter accept comment "proxy-web"

# logging
add rule ip  filter FORWARD-CB iifname $IF-INET counter log prefix "nft-drop-DOCKER-USER: " flags tcp options flags ip options
add rule ip6 filter FORWARD-CB iifname $IF-INET counter log prefix "nft-drop-DOCKER-USER: " flags tcp options flags ip options

# drop all
add rule ip  filter FORWARD-CB iifname $IF-INET counter drop comment "drop-all"
add rule ip6 filter FORWARD-CB iifname $IF-INET counter drop comment "drop-all"

# return to FORWARD chain
add rule ip  filter FORWARD-CB counter return comment "return to DOCKER-USER chain"
add rule ip6 filter FORWARD-CB counter return comment "return to DOCKER-USER chain"

[Keonik1]

Since you're mentioning iptables-nft in the Additional Info section, could you try to switch to iptables-legacy with the following commands:

# Select the legacy option each time
$ update-alternatives --config iptables
$ update-alternatives --config ip6tables
# Then:
$ sudo systemctl restart docker.service

link to the referenced comment

UPD:

What I did, did not help, or rather, if I create new containers, they do not work, if I restart nftables.service, then everything stops working.

Something that helped me

In my case i switched FROM iptables-legacy, to iptables-nft and now docker is working ?fine? (may be something is broken, but I haven't noticed it at the moment)

Additional info:

I updated from debian 11.7 -> 11.9 to debian 12.5.
I have a backup of debian 11 before the update and the iptables-nft parameter is there, I do not know why it suddenly became iptables-legacy after the system upgrade.

Versions:

# cat /etc/debian_version 
12.5
# docker version
Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:01 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
# sudo iptables --version
iptables v1.8.9 (nf_tables)
# sudo LANG=en update-alternatives --config iptables
There are 2 choices for the alternative iptables (providing /usr/sbin/iptables).

  Selection    Path                       Priority   Status
------------------------------------------------------------
  0            /usr/sbin/iptables-nft      20        auto mode
* 1            /usr/sbin/iptables-legacy   10        manual mode
  2            /usr/sbin/iptables-nft      20        manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
# sudo LANG=en update-alternatives --config ip6tables
There are 2 choices for the alternative ip6tables (providing /usr/sbin/ip6tables).

  Selection    Path                        Priority   Status
------------------------------------------------------------
  0            /usr/sbin/ip6tables-nft      20        auto mode
* 1            /usr/sbin/ip6tables-legacy   10        manual mode
  2            /usr/sbin/ip6tables-nft      20        manual mode

Press <enter> to keep the current choice[*], or type selection number: 2

Possible fix

switch the current iptables to iptables-legacy or iptables-nft, maybe it will work in your case.

That I tried and it didn't help me

1. I tried to upgrade iptables to version 1.8.10-3
2. I tried downgrade iptables to version 1.8.7-1

Neither the first nor the second helped me, while the second also broke the network - I had to roll back and reboot.

3. Turning off nftables helped, but it's not my option.

[Keonik1]

tl;dr

sudo update-alternatives --remove iptables /usr/sbin/iptables-legacy
sudo update-alternatives --remove ip6tables /usr/sbin/ip6tables-legacy
sudo systemctl enable nftables.service #if disabled
sudo shutdown -r now

Description

Okay, I spent a little more time (a lot of time...).

In my case, the problem was an incomprehensible bug, when performing update-alternatives and rebooting the system, in fact, the selected parameter was reset back to iptables-legacy...
The bug appeared after upgrading the system from debian 11 to debian 12.

Therefore, I removed iptables-legacy from the alternatives altogether and everything became normal.

Others

# cat /etc/nftables.conf
#!/usr/sbin/nft -f

flush ruleset

# your rules
# your rules
# your rules

# cat /etc/debian_version 
12.5

# uname -a
Linux xxxxxxxx 6.5.13-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-3 (2024-03-20T10:45Z) x86_64 GNU/Linux

# sudo iptables --version
iptables v1.8.9 (nf_tables)

# sudo nft --version
nftables v1.0.6 (Lester Gooch #5)

# docker version
Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:01 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

# sudo update-alternatives --list iptables
/usr/sbin/iptables-nft