lkml.org 
[lkml]   [2022]   [Sep]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subject[REGRESSION] Unable to NAT own TCP packets from another VRF with tcp_l3mdev_accept = 1
Hello,

We're using VRF with a machine used as a router and have a specific
issue where the router doesn't handle his own packets correctly during
NATing if the packet is coming from a different VRF.

We had the issue with debian buster (4.19), but the issue solved itself
when we updated to debian bullseye (5.10.92).

However, during an upgrade of debian bullseye to the latest kernel, the
issue appeared again (5.10.140).

We did a bisection and this leaded us to
"b0d67ef5b43aedbb558b9def2da5b4fffeb19966 net: allow unbound socket for
packets in VRF when tcp_l3mdev_accept set [ Upstream commit
944fd1aeacb627fa617f85f8e5a34f7ae8ea4d8e ]".

Simplified case setup:

There is two machines in the setup. They both forward packets
(net.ipv4.ip_forward = 1) and there is two interface between them.

The main machine has two VRF. The default VRF is using the second
machine as the default route, on a specific interface.
The second machine has as default route to main machine, on the other
VRF using the second pair of interfaces.

On the main machine, the second interface is in a specific VRF. In that
VRF, packets are NATed to the internet on a third interface.

A visual schema with the normal flow is available there:
https://etinacra.ch/kernel.png

Configuration command:

Main machine:
sysctl -w net.ipv4.tcp_l3mdev_accept = 1
sysctl -w systnet.ipv4.ip_forward = 1
iptables -t raw -A PREROUTING -i eth0 -j CT --zone 5
iptables -t raw -A OUTPUT -o eth0 -j CT --zone 5
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to 192.168.1.1
cat /etc/network/interfaces

auto firewall
iface firewall
    vrf-table 1200

auto eth0
iface eth0
    address 192.168.5.1/24
    gateway 192.168.5.2

auto eth1
iface eth1
    address 192.168.10.1/24
    vrf firewall
    up ip route add 192.168.5.0/24 via 192.168.10.2 vrf firewall

auto eth2
iface eth2
    address 192.168.1.1/24
    gateway 192.168.1.250
    vrf firewall

==

Second machine:

sysctl -w net.ipv4.ip_forward = 1

cat /etc/network/interfaces

auto eth0
iface eth0
    address 192.168.5.2/24

auto eth1
iface eth1
    address 192.168.10.2/24
    gateway 192.168.10.1

==

Without issue, if we look at a tcpdump on all interface on the main
machine, everything is fine (output truncated):

10:28:32.811283 eth0 Out IP 192.168.5.1.55750 > 99.99.99.99.80: Flags
[S], seq 2216112145
10:28:32.811666 eth1 In  IP 192.168.5.1.55750 > 99.99.99.99.80: Flags
[S], seq 2216112145
10:28:32.811679 eth2 Out IP 192.168.1.1.55750 > 99.99.99.99.80: Flags
[S], seq 2216112145
10:28:32.835138 eth2 In  IP 99.99.99.99.80 > 192.168.1.1.55750: Flags
[S.], seq 383992840, ack 2216112146
10:28:32.835152 eth1 Out IP 99.99.99.99.80 > 192.168.5.1.55750: Flags
[S.], seq 383992840, ack 2216112146
10:28:32.835457 eth0 In  IP 99.99.99.99.80 > 192.168.5.1.55750: Flags
[S.], seq 383992840, ack 2216112146
10:28:32.835511 eth0 Out IP 192.168.5.1.55750 > 99.99.99.99.80: Flags
[.], ack 1, win 502

However when the issue is present, the SYNACK does arrives on eth2, but
is never "unNATed" back to eth1:

10:25:07.644433 eth0 Out IP 192.168.5.1.48684 > 99.99.99.99.80: Flags
[S], seq 3207393154
10:25:07.644782 eth1 In  IP 192.168.5.1.48684 > 99.99.99.99.80: Flags
[S], seq 3207393154
10:25:07.644793 eth2 Out IP 192.168.1.1.48684 > 99.99.99.99.80: Flags
[S], seq 3207393154
10:25:07.668551 eth2 In  IP 54.36.61.42.80 > 192.168.1.1.48684: Flags
[S.], seq 823335485, ack 3207393155

The issue is only with TCP connections. UDP or ICMP works fine.

Turing off net.ipv4.tcp_l3mdev_accept back to 0 also fix the issue, but
we need this flag since we use some sockets that does not understand VRFs.

We did have a look at the diff and the code of inet_bound_dev_eq, but we
didn't understand much the real problem - but it does seem now that
bound_dev_if if now checked not to be False before the bound_dev_if ==
dif || bound_dev_if == sdif comparison, something that was not the case
before (especially since it's dependent on l3mdev_accept).

Maybe our setup is wrong and we should not be able to route packets like
that?

Thanks a lot and have a nice day!

Maximilien Cuony


\
 
 \ /
  Last update: 2022-09-28 16:29    [W:0.029 / U:4.076 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site