5G SCTP LoadBalancer Using LoxiLB
This is the second blog post that I will be writing about LoxiLB and this one focuses on how to use LoxiLB as an SCTP loadbalancer to loadbalance N2 traffic between 2 or more AMFs.
Git repo for this blog-post: https://github.com/infinitydon/5gcore-sctp-loadbalancer
The git repo has all the manifests file that was used in creating the deployments.
The architecture is shown below:
A kubernetes cluster is used to host all the functions so as to make it easier to experiment. This should be able to give clearer picture if it is required to deploy any of the components in a stand-alone fashion.
Below shows the description of the components that will be used:
- 5G Core: Open5gs will be used.
- UE/gNB: my5gRANTester
- SCTP LoadBalancer: LoxiLB, this will be deployed within the Kubernetes cluster as a statefulset
- Kubernetes Cluster: KIND
- Multus: This is used for the traffic separation within the Kubernetes cluster, this also provides static IPs that can be used as endpoints for the SCTP configuration in the LoxiLB
- koko: Provides the ability to create/inject additional interfaces inside docker containers, the interfaces created will be used by Multus to create secondary network interfaces in the AMF, SMF, UPF, LoxiLB and my5gRANTester
The load-balancing scheme that was used in the LoxiLB is onearm, in this mode LoxiLB will replace source IP from the VIP (20.20.20.1) to it’s outgoing IP (10.0.3.1). The other mode is fullnat, in this mode the VIP is not changed (i.e. AMFs will see traffic coming from VIP itself), this will require proper routing is configured to cater for the VIP traffic (LoxiLB should be used as the gateway for the VIP in the gNB/AMFs).
Summary of the IP setup (emphasis on the Multus secondary interfaces):
- SCTP VIP that is configured in LoxiLB: 20.20.20.1
- my5gRANTester Multus Interface: 10.0.3.2/24
- LoxiLB Multus Interface: 10.0.3.1/24
- AMF-1 Multus Interface: 10.0.3.3/24
- AMF-2 Multus Interface: 10.0.3.4/24
- SMF Multus Interface: 10.0.3.5/24
- UPF Multus Interface: 10.0.3.6/24
LoxiLB command to initiate the LB:
loxicmd create lb 20.20.20.1 --sctp=38412:38412 --endpoints=10.0.3.3:1,10.0.3.4:1 --mode=onearmStatus of the Kubernetes deployments
root@ebenezer-k8s-loxi:~# kubectl -n open5gs get po
NAME READY STATUS RESTARTS AGE
core5g-amf-1-deployment-59b6df76fb-w26lz 1/1 Running 0 25m
core5g-amf-2-deployment-57688d5cff-p6w6j 1/1 Running 0 25m
core5g-ausf-deployment-679587ff99-b8trs 1/1 Running 0 25m
core5g-bsf-deployment-7b878b647-d5dk8 1/1 Running 0 25m
core5g-mongo-ue-import-hzxrl 0/1 Completed 0 25m
core5g-mongodb-6f9f5c7f9f-jblt7 1/1 Running 0 25m
core5g-nrf-deployment-b566fd6f9-kffrc 1/1 Running 0 25m
core5g-nssf-deployment-d8574cd6-x4bzn 1/1 Running 0 25m
core5g-pcf-deployment-bb4bb46d5-sfw5p 1/1 Running 1 (24m ago) 25m
core5g-smf-deployment-6bdd5c97fb-xw92b 1/1 Running 0 25m
core5g-udm-deployment-64469df976-twchh 1/1 Running 0 25m
core5g-udr-deployment-7d464868f7-lnkfq 1/1 Running 0 25m
core5g-upf-deployment-8645cdfcb5-44pl9 2/2 Running 0 25m
core5g-webui-66d9b59d74-wwmtl 1/1 Running 0 25mroot@ebenezer-k8s-loxi:~# kubectl -n loadbalancer get po
NAME READY STATUS RESTARTS AGE
lb5g-0 1/1 Running 0 23h
root@ebenezer-k8s-loxi:~#
root@ebenezer-k8s-loxi:~#
root@ebenezer-k8s-loxi:~# kubectl -n ran-simulator get po
NAME READY STATUS RESTARTS AGE
sim5g-simulator-8b7bc977d-8b4cm 1/1 Running 0 23hLoxiLB statistics:
root@ebenezer-k8s-loxi:~# kubectl -n loadbalancer exec -ti lb5g-0 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@lb5g-0:/#
root@lb5g-0:/#
root@lb5g-0:/# loxicmd get ct
| DESTINATIONIP | SOURCEIP | DESTINATIONPORT | SOURCEPORT | PROTOCOL | STATE | ACT | PACKETS | BYTES |
|---------------|----------|-----------------|------------|----------|-------|------------------------------------|---------|-------|
| 10.0.3.1 | 10.0.3.4 | 9487 | 38412 | sctp | est | fsnat-20.20.20.1,10.0.3.2:38412:w0 | 105 | 11378 |
| 20.20.20.1 | 10.0.3.2 | 38412 | 9487 | sctp | est | fdnat-10.0.3.1,10.0.3.4:38412:w0 | 105 | 11314 |my5gRANTester:
kubectl -n ran-simulator exec -ti deploy/sim5g-simulator bash
./app load-test -n 1
root@sim5g-simulator-8b7bc977d-8b4cm:/workspace/my5G-RANTester/cmd# ./app load-test -n 1
INFO[0000] my5G-RANTester version 1.0.1
INFO[0000] ---------------------------------------
INFO[0000] [TESTER] Starting test function: Testing registration of multiple UEs
INFO[0000] [TESTER][UE] Number of UEs: 1
INFO[0000] [TESTER][GNB] gNodeB control interface IP/Port: 10.0.3.2/9487
INFO[0000] [TESTER][GNB] gNodeB data interface IP/Port: 10.0.3.2/2152
INFO[0000] [TESTER][AMF] AMF IP/Port: 20.20.20.1/38412
INFO[0000] ---------------------------------------
INFO[0000] [GNB] SCTP/NGAP service is running
INFO[0000] [GNB] UNIX/NAS service is running
encoding/hex: odd length hex string
INFO[0000] [GNB][SCTP] Receive message in 0 stream
INFO[0000] [GNB][NGAP] Receive Ng Setup Response
INFO[0000] [GNB][AMF] AMF Name: open5gs-amf2
INFO[0000] [GNB][AMF] State of AMF: Active
INFO[0000] [GNB][AMF] Capacity of AMF: 255
INFO[0000] [GNB][AMF] PLMNs Identities Supported by AMF -- mcc: 208 mnc:93
INFO[0000] [GNB][AMF] List of AMF slices Supported by AMF -- sst:01 sd:000001
INFO[0001] [TESTER] TESTING REGISTRATION USING IMSI 0000000031 UE
INFO[0001] [UE] UNIX/NAS service is running
INFO[0001] [GNB][SCTP] Receive message in 1 stream
INFO[0001] [GNB][NGAP] Receive Downlink NAS Transport
INFO[0001] [UE][NAS] Message without security header
INFO[0001] [UE][NAS] Receive Authentication Request
INFO[0001] [UE][NAS][MAC] Authenticity of the authentication request message: OK
INFO[0001] [UE][NAS][SQN] SQN of the authentication request message: VALID
INFO[0001] [UE][NAS] Send authentication response
INFO[0001] [GNB][SCTP] Receive message in 1 stream
INFO[0001] [GNB][NGAP] Receive Downlink NAS TransportFrom the above, LoxiLB is using the AMF-2 for the N2 association, we can double-check this from the AMF-2 logs:
12/07 01:13:54.319: [sbi] INFO: NF Service [namf-comm] (../lib/sbi/context.c:1408)
12/07 01:13:54.319: [sbi] INFO: nghttp2_server() [0.0.0.0]:80 (../lib/sbi/nghttp2-server.c:150)
12/07 01:13:54.320: [amf] INFO: ngap_server() [10.0.3.4]:38412 (../src/amf/ngap-sctp.c:61)
12/07 01:13:54.323: [sctp] INFO: AMF initialize...done (../src/amf/app.c:33)
12/07 01:13:54.334: [sbi] INFO: [6abafc5c-75cc-41ed-befa-55641de75816] NF registered [Heartbeat:10s] (../lib/sbi/nf-sm.c:214)
12/07 01:19:07.691: [amf] INFO: gNB-N2 accepted[10.0.3.1]:9487 in ng-path module (../src/amf/ngap-sctp.c:113)
12/07 01:19:07.694: [amf] INFO: gNB-N2 accepted[10.0.3.1] in master_sm module (../src/amf/amf-sm.c:668)
12/07 01:19:07.698: [amf] INFO: [Added] Number of gNBs is now 1 (../src/amf/context.c:973)
12/07 01:19:07.698: [amf] INFO: gNB-N2[10.0.3.1] max_num_of_ostreams : 2 (../src/amf/amf-sm.c:707)
12/07 01:19:08.694: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:361)
12/07 01:19:08.694: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2239)
12/07 01:19:08.694: [amf] INFO: RAN_UE_NGAP_ID[1] AMF_UE_NGAP_ID[1] TAC[7] CellID[0x1] (../src/amf/ngap-handler.c:503)
12/07 01:19:08.694: [amf] INFO: [suci-0-208-93-0-0-0-0000000031] Unknown UE by SUCI (../src/amf/context.c:1546)
12/07 01:19:08.694: [amf] INFO: [Added] Number of AMF-UEs is now 1 (../src/amf/context.c:1340)
12/07 01:19:08.694: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:135)
12/07 01:19:08.694: [gmm] INFO: [suci-0-208-93-0-0-0-0000000031] SUCI (../src/amf/gmm-handler.c:149)The AMF is seeing traffic from the LoxiLB multus interface IP gNB-N2 accepted[10.0.3.1]:9487
There are other cool features of LoxiLB but I think this blog post should already give you more hints about the possible scenarios that you can build for your 5G network architecture.
I will like to thank the LoxiLB developers for patiently answering my questions and guidance where needed.
Thanks to Tomofumi Hayashi for his github gist, this greatly made it easier to setup Multus on a KIND kubernetes cluster.
P.S — If you want to learn more about Kubernetes networking, you can take a look at my Udemy course: https://www.udemy.com/course/kubernetes-baremetal-networking-using-gns3
