5G Core SBI mTLS Using External Certificate PKI
The 5G core SBI uses HTTP2 protocol for communication and from a security perspective this communication channel should be secured.
One of the ways of securing the HTTP2 SBI traffic is to make use of mTLS certificates, while it is possible to directly embed the certificate in the container images, this is not recommended because if the certificate needs to be rotated then this means that the container images will have to be re-built and then the containers must be re-created. Usually certificate management should be offloaded to a separate function like a central PKI solution.
An example of an opensource PKI solution is SmallStep CA and they also have a related project called AUTOCERT that can inject mTLS certificates to Kubernetes PODs via a side-car. The side-car container is responsible for updating the certificates automatically in case there is need for the certificate to be rotated.
In this post, I will be using Open5gs to show how this can be done. With the latest releases of Open5gs, you can configure specific certificate locations (root.crt, site.crt and site.crt) that will be used to encrypt the SBI communication.
Below shows the design architecture
In order to inject the TLS certificates into the respective PODs, an annotation is simply added to the manifest:
annotations:
autocert.step.sm/name: core5g-amf.open5gs.svc.cluster.local
autocert.step.sm/duration: 30mWhat this means is that the domain name that will be used for generating the certificate for the AMF is core5g-amf.open5gs.svc.cluster.local and after 30 minutes the certificate will be rotated automatically.
Status of the PODs after deployment:
kubectl -n open5gs get po
NAME READY STATUS RESTARTS AGE
core5g-amf-deployment-544667f9c7-z4bj6 2/2 Running 0 42m
core5g-ausf-deployment-668fb8d8f8-zn628 2/2 Running 0 42m
core5g-bsf-deployment-57fcfb768d-rlffq 2/2 Running 0 42m
core5g-mongodb-5c8964b55d-7wmnc 1/1 Running 0 42m
core5g-nrf-deployment-65db5975b5-b79jm 2/2 Running 0 42m
core5g-nssf-deployment-7956d984d4-67sdz 2/2 Running 0 42m
core5g-pcf-deployment-d76fcd97-bq9hv 2/2 Running 0 42m
core5g-scp-deployment-cdb486c95-x6tml 2/2 Running 0 42m
core5g-smf-deployment-8474b579df-5fg5g 2/2 Running 0 42m
core5g-udm-deployment-848c5bf848-xx2jc 2/2 Running 0 42m
core5g-udr-deployment-5797cc596f-g2rmw 2/2 Running 0 42m
core5g-ue-import-8556598584-k4gt4 1/1 Running 0 42m
core5g-ueransim-gnb-7bcd5cfd6c-jbnqg 1/1 Running 0 42m
core5g-ueransim-ue-78ddfb8874-sf5pm 1/1 Running 0 42m
core5g-upf-deployment-696f8c6597-22kkp 2/2 Running 0 42m
core5g-webui-b44ddc56-rvm7d 1/1 Running 0 42mEach of the NFs that uses SBI to communicate has 2 containers running, one for the NF itself and the other for the SmallStep CA autocert side-car.
We can exec into the AMF pod and try to curl to the NRF to check if indeed the communication is been encrypted using the SmallStep certificate:
kubectl -n open5gs exec -ti deploy/core5g-amf-deployment -c amf bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@core5g-amf-deployment-544667f9c7-z4bj6:/#curl -v \
--http2-prior-knowledge --cacert /var/run/autocert.step.sm/root.crt \
--cert /var/run/autocert.step.sm/site.crt \
--key /var/run/autocert.step.sm/site.key \
https://core5g-nrf.open5gs.svc.cluster.local/nnrf-nfm/v1/nf-instances
* Trying 10.96.196.11:443...
* Connected to core5g-nrf.open5gs.svc.cluster.local (10.96.196.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /var/run/autocert.step.sm/root.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=core5g-nrf.open5gs.svc.cluster.local
* start date: Apr 17 18:22:30 2023 GMT
* expire date: Apr 17 18:53:30 2023 GMT
* subjectAltName: host "core5g-nrf.open5gs.svc.cluster.local" matched cert's "core5g-nrf.open5gs.svc.cluster.local"
* issuer: O=Step Certificates; CN=Step Certificates Intermediate CA
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5569c82cee90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /nnrf-nfm/v1/nf-instances HTTP/2
> Host: core5g-nrf.open5gs.svc.cluster.local
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 16384)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< server: Open5GS v2.6.2+
< date: Mon, 17 Apr 2023 18:24:05 GMT
< content-length: 938
< content-type: application/3gppHal+json
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{
"_links": {
"items": [{
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/f4dd9c4a-dd4c-41ed-9116-2916c32a6bf4"
}, {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/8aaaa7a6-dd3c-41ed-b3ac-dd16c22c3ff8"
}, {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/8db82d10-dd3c-41ed-b2a9-8f1ca59a19f8"
}, {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/8dad45a8-dd3c-41ed-8756-57169af3fdd6"
}, {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/8db091a4-dd3c-41ed-9293-e98ef8e5a5ee"
}, {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/8e49bc8a-dd3c-41ed-a336-3125c5b98d16"
}, {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/8e48528c-dd3c-41ed-a7a1-054a0146cb62"
}, {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances/8e490646-dd3c-41ed-add8-ff54e240ca2f"
}],
"self": {
"href": "https://10.244.1.54/nnrf-nfm/v1/nf-instances"
}
}
* Connection #0 to host core5g-nrf.open5gs.svc.cluster.local left intactPart of the outputs that is interesting:
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=core5g-nrf.open5gs.svc.cluster.local
* start date: Apr 17 18:22:30 2023 GMT
* expire date: Apr 17 18:53:30 2023 GMT
* subjectAltName: host “core5g-nrf.open5gs.svc.cluster.local” matched cert’s “core5g-nrf.open5gs.svc.cluster.local”
* issuer: O=Step Certificates; CN=Step Certificates Intermediate CA
* SSL certificate verify ok.
The SmallStep statefulset can also be checked to confirm that is creating the required certificate:
17T07:06:30Z” issuer=”Step Certificates Intermediate CA” method=POST name=ca path=/renew protocol=HTTP/1.1 provisioner=”admin (zIc-iIsum3V_MS3bbJaKuKbxiTDLpWrb8nLiL6HODS8)” public-key=”ECDSA P-256" referer= remote-address=10.244.1.224 request-id=cguevtiev3074s0d242g serial=74216305176415804391374177547331512835
size=3513 status=201 subject=core5g-amf.open5gs.svc.cluster.local user-agent=Go-http-client/1.1 user-id= valid-from=”2023–04–17T07:05:30Z” valid-to=”2023–04–17T07:36:30Z”
We can see that the subject CN matches that of the AMF and also the certificate will expire in 30 minutes after which it will be replaced by the SmallStep autocert side-car.
The manifest files that was used for this post can be found at:
I hope you find this interesting as I did. One thing to take note is that since Open5gs does not support hot-reloading, you will need to use another utility like inotify to restart the process once the certificates has been rotated.
Thanks to Sukchan Lee for fixing the bug I encountered in Open5gs when I was creating this deployment.
References:
