Monthly Archives: December 2022

AKS Private Cluster, kubenet, UDR, and Azure Firewall

Azure, Networking, Security, Workshop

This configuration will be covered the following requirement:

  • AKS Private Cluster with kubenet
  • Bring your own subnet with route table
  • limit egress with Azure Firewall
  • Access to ACR via public endpoint
  • Todo:
    • Access to ACR via private endpoint
    • Access to Azure Database for PostgreSQL via private endpoint
    • Monitoring via private link
    • Traffic flow query
    • Adding Azure Front Door/Application Gateway for public access

Private-AKS-AFW-egress-UDR
Private-AKS-AFW-egress-UDR

Creating AKS Private Cluster with Azure CLI

Sample output

a@myVM001:~$ ip address | grep "inet 10.42"
    inet 10.42.3.4/24 metric 100 brd 10.42.3.255 scope global eth0
a@myVM001:~$ kubectl get pods,svc
NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-676579fdc-68b6d   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-7kdwg   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-q7q9f   1/1     Running   0          6m30s

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/fuju-nginx   LoadBalancer   10.0.141.238   10.42.1.7     80:31373/TCP   6m30s
service/kubernetes   ClusterIP      10.0.0.1       <none>        443/TCP        63m
a@myVM001:~$ curl -s http://10.42.1.7/ | grep title
<title>Welcome to nginx!</title>
a@myVM001:~$

Related Error Messages:

The following is “Error messages” when AKS Node unable to access ACR 

a@myVM001:~$ az aks check-acr --resource-group $RG --name $AKSNAME --acr $MYACR.azurecr.io
Merged "aks-egress" as current context in /tmp/tmpaacrqofd
WARNING: version difference between client (1.26) and server (1.23) exceeds the supported minor version skew of +/-1
[2022-12-11T09:27:52Z] Checking host name resolution ($MYACR.azurecr.io): SUCCEEDED
[2022-12-11T09:27:52Z] Canonical name for ACR ($MYACR.azurecr.io): r0000ea.eastasia.cloudapp.azure.com.
[2022-12-11T09:27:52Z] ACR location: eastasia
[2022-12-11T09:27:52Z] Checking managed identity...
[2022-12-11T09:27:52Z] Kubelet managed identity client ID: 000-0000-0000-0000-0000c
[2022-12-11T09:27:53Z] Validating managed identity existance: SUCCEEDED
[2022-12-11T09:27:53Z] Validating image pull permission: FAILED
[2022-12-11T09:27:53Z] ACR myacregistry001.azurecr.io rejected token exchange: failed to send token exchange request: Post "https://$MYACR.azurecr.io/oauth2/exchange": EOF

The following is “Error message” when creating AKS cluster but $IDENTITY_ID doesn’t have permission to create entry in route table (UDR) – Additional information for built-in-roles (Network Contributor) can be found @ https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#network-contributor. And ” Microsoft.Network/routeTables/*” provider operations can be found details under https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftnetwork ” 

 (CustomRouteTableMissingPermission) Managed identity or service principle must be given permission to read and write to custom route table /subscriptions/0000-0000-0000-0000-300a/resourceGroups/101-aks-egress-rg/providers/Microsoft.Network/routeTables/aks-egress-fwrt. Please see https://aka.ms/aks/customrt for more information

Sample Route Table after adding AKS ( Kubenet ) in VNET

Sample Route Table

See Also: