5G Frame Routing

One of the interesting features of 5G is the ability to provide connectivity from external N6 networks to IP networks that are sitting behind a UE (technically we can call this type of UE a router or CPE).

This feature is called Frame Routing whereby another IP subnet is associated with a UE alongside the normal IP address that is assigned from the UE IP pool. You can check 3GPP TS 23.501 section 5.6.14 Support of Framed Routing for more details.

Ever since this feature has been added to Open5gs, I have been wanting to try this but getting it to work in a simple fashion whereby others can also replicate easily is what requires a bit of effort. Luckily I stumbled upon Shigeru Ishida post on this.

What I have done in this post is:

1. To move the whole architecture to Kubernetes using KIND
2. Use combination of koko and multus to fully simulate the end to end networks
3. Instead of using MongoDB Compass, I modified the open5gs-dbctl script to support frame routes for the UE subscription in the Mongo DB. I created specific templates using configmaps for the UEs.

Architecture is shown below

As shown above, the quest is for the external Test Node to communicate via the UPF to the internal networks that UE-1 and UE-2 are connected to.

The templates that were used for this post cnab found in my github repo:

https://github.com/infinitydon/5g-framed-routing

Below is the snippet of what will be created in the subscriber database:

UE-1:

UE-2:

To test the connectivity and also ensure that the traffic destined for UE-1 internal is not routed to UE-2, we can initiate a ping test from the external test node to UE-1 internal network and run tcpdump on UE-2 while the ping is ongoing.

Ping test from external test node:

root@o5gs-ext-test-node-5dc8695554-m9sw8:/# ping 192.168.20.100 -c4
PING 192.168.20.100 (192.168.20.100): 56 data bytes
64 bytes from 192.168.20.100: icmp_seq=0 ttl=63 time=0.689 ms
64 bytes from 192.168.20.100: icmp_seq=1 ttl=63 time=0.645 ms
64 bytes from 192.168.20.100: icmp_seq=2 ttl=63 time=0.636 ms
64 bytes from 192.168.20.100: icmp_seq=3 ttl=63 time=0.628 ms
--- 192.168.20.100 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.628/0.650/0.689/0.024 ms
root@o5gs-ext-test-node-5dc8695554-m9sw8:/#

TCPDUMP on UE-1:

root@o5gs-ueransim-ue-1-5f94898995-qr929:/UERANSIM/build# tcpdump -i any host 172.16.100.2 -s 0 -nv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
13:47:00.458696 IP (tos 0x0, ttl 63, id 38726, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.100.2 > 192.168.20.100: ICMP echo request, id 530, seq 0, length 64
13:47:00.458718 IP (tos 0x0, ttl 64, id 19601, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.100 > 172.16.100.2: ICMP echo reply, id 530, seq 0, length 64
13:47:01.459669 IP (tos 0x0, ttl 63, id 38729, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.100.2 > 192.168.20.100: ICMP echo request, id 530, seq 1, length 64
13:47:01.459692 IP (tos 0x0, ttl 64, id 19838, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.100 > 172.16.100.2: ICMP echo reply, id 530, seq 1, length 64
13:47:02.460734 IP (tos 0x0, ttl 63, id 38757, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.100.2 > 192.168.20.100: ICMP echo request, id 530, seq 2, length 64
13:47:02.460753 IP (tos 0x0, ttl 64, id 19920, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.100 > 172.16.100.2: ICMP echo reply, id 530, seq 2, length 64
13:47:03.461784 IP (tos 0x0, ttl 63, id 38866, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.100.2 > 192.168.20.100: ICMP echo request, id 530, seq 3, length 64
13:47:03.461806 IP (tos 0x0, ttl 64, id 19988, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.100 > 172.16.100.2: ICMP echo reply, id 530, seq 3, length 64

^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

TCPDUMP on UE-2:

root@o5gs-ueransim-ue-2-5cc97878bd-qxh8h:/UERANSIM/build# tcpdump -i any host 172.16.100.2 -s 0 -nv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

0 packets captured
0 packets received by filter
0 packets dropped by kernel

As per the routing config, following should be noted:

1. External test node must use the UPF as the next-hop for the UEs’ internal networks
2. The UPF in turn must use the tunnel interface as the next-hop for the UEs’ internal networks
3. The UEs must use their tunnel interface as the next-hop for the external test node return traffic

External test node routing table:

root@o5gs-ext-test-node-5dc8695554-m9sw8:/# ip r
default via 10.244.1.1 dev eth0
10.244.1.0/24 via 10.244.1.1 dev eth0 src 10.244.1.35
10.244.1.1 dev eth0 scope link src 10.244.1.35
172.16.100.0/24 dev net1 proto kernel scope link src 172.16.100.2
192.168.20.0/24 via 172.16.100.1 dev net1
192.168.21.0/24 via 172.16.100.1 dev net1

UPF routing table:

root@o5gs-upf-deployment-5c9956c789-rjlpn:/# ip r
default via 10.244.1.1 dev eth0
10.45.0.0/16 dev ogstun proto kernel scope link src 10.45.0.1
10.244.1.0/24 via 10.244.1.1 dev eth0 src 10.244.1.38
10.244.1.1 dev eth0 scope link src 10.244.1.38
172.16.100.0/24 dev net1 proto kernel scope link src 172.16.100.1
192.168.20.0/24 dev ogstun scope link
192.168.21.0/24 dev ogstun scope link

UE-1 routing table:

root@o5gs-ueransim-ue-1-5f94898995-qr929:/UERANSIM/build# ip r
default via 10.244.2.1 dev eth0
10.45.0.0/16 dev uesimtun0 scope link
10.244.2.0/24 via 10.244.2.1 dev eth0 src 10.244.2.48
10.244.2.1 dev eth0 scope link src 10.244.2.48
172.16.100.0/24 dev uesimtun0 scope link
192.168.20.0/24 dev net1 proto kernel scope link src 192.168.20.100

UE-2 routing table:

root@o5gs-ueransim-ue-2-5cc97878bd-qxh8h:/UERANSIM/build# ip r
default via 10.244.2.1 dev eth0
10.45.0.0/16 dev uesimtun0 scope link
10.244.2.0/24 via 10.244.2.1 dev eth0 src 10.244.2.46
10.244.2.1 dev eth0 scope link src 10.244.2.46
172.16.100.0/24 dev uesimtun0 scope link
192.168.21.0/24 dev net1 proto kernel scope link src 192.168.21.100

References

Framed Routing in 5G

GitHub - s5uishida/open5gs_5gc_ueransim_framed_routing_sample_config: Open5GS 5GC & UERANSIM UE /…