This configuration will be covered the following requirement:
- AKS Private Cluster with kubenet
- Bring your own subnet with route table
- limit egress with Azure Firewall
- Access to ACR via public endpoint
- Todo:
- Access to ACR via private endpoint
- Access to Azure Database for PostgreSQL via private endpoint
- Monitoring via private link
- Traffic flow query
- Adding Azure Front Door/Application Gateway for public access

Creating AKS Private Cluster with Azure CLI
Sample output
a@myVM001:~$ ip address | grep "inet 10.42" inet 10.42.3.4/24 metric 100 brd 10.42.3.255 scope global eth0 a@myVM001:~$ kubectl get pods,svc NAME READY STATUS RESTARTS AGE pod/nginx-deployment-676579fdc-68b6d 1/1 Running 0 6m30s pod/nginx-deployment-676579fdc-7kdwg 1/1 Running 0 6m30s pod/nginx-deployment-676579fdc-q7q9f 1/1 Running 0 6m30s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/fuju-nginx LoadBalancer 10.0.141.238 10.42.1.7 80:31373/TCP 6m30s service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 63m a@myVM001:~$ curl -s http://10.42.1.7/ | grep title <title>Welcome to nginx!</title> a@myVM001:~$
Related Error Messages:
The following is “Error messages” when AKS Node unable to access ACR
a@myVM001:~$ az aks check-acr --resource-group $RG --name $AKSNAME --acr $MYACR.azurecr.io Merged "aks-egress" as current context in /tmp/tmpaacrqofd WARNING: version difference between client (1.26) and server (1.23) exceeds the supported minor version skew of +/-1 [2022-12-11T09:27:52Z] Checking host name resolution ($MYACR.azurecr.io): SUCCEEDED [2022-12-11T09:27:52Z] Canonical name for ACR ($MYACR.azurecr.io): r0000ea.eastasia.cloudapp.azure.com. [2022-12-11T09:27:52Z] ACR location: eastasia [2022-12-11T09:27:52Z] Checking managed identity... [2022-12-11T09:27:52Z] Kubelet managed identity client ID: 000-0000-0000-0000-0000c [2022-12-11T09:27:53Z] Validating managed identity existance: SUCCEEDED [2022-12-11T09:27:53Z] Validating image pull permission: FAILED [2022-12-11T09:27:53Z] ACR myacregistry001.azurecr.io rejected token exchange: failed to send token exchange request: Post "https://$MYACR.azurecr.io/oauth2/exchange": EOF
The following is “Error message” when creating AKS cluster but $IDENTITY_ID doesn’t have permission to create entry in route table (UDR) – Additional information for built-in-roles (Network Contributor) can be found @ https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#network-contributor. And ” Microsoft.Network/routeTables/*” provider operations can be found details under https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftnetwork ”
(CustomRouteTableMissingPermission) Managed identity or service principle must be given permission to read and write to custom route table /subscriptions/0000-0000-0000-0000-300a/resourceGroups/101-aks-egress-rg/providers/Microsoft.Network/routeTables/aks-egress-fwrt. Please see https://aka.ms/aks/customrt for more information
Sample Route Table after adding AKS ( Kubenet ) in VNET

See Also:
- limit egress traffic
- https://learn.microsoft.com/en-us/azure/aks/configure-kubenet#bring-your-own-subnet-and-route-table-with-kubenet
- CyberSecurity
- Intergration
- Firewall related
- https://learn.microsoft.com/en-us/azure/container-registry/container-registry-firewall-access-rules
- https://github.com/microsoft/containerregistry/blob/main/client-firewall-rules.md
- https://learn.microsoft.com/en-us/services-hub/health/assessments-troubleshooting
- https://learn.microsoft.com/en-us/azure/architecture/example-scenario/aks-firewall/aks-firewall
- https://www.microsoft.com/en-us/download/details.aspx?id=56519
- {Workspace GUID}.ods.opinsights.azure.com
- {Workspace GUID}.oms.opinsights.azure.com
- Choosing a network model to use
- Baseline
- IP Ranges
- Azure IP Ranges and Service Tags – Public Cloud : https://www.microsoft.com/en-us/download/details.aspx?id=56519
- https://learn.microsoft.com/en-us/azure/azure-portal/azure-portal-safelist-urls?tabs=public-cloud
- https://learn.microsoft.com/en-us/answers/questions/421629/whitelisted-url-to-enable-user-sign-in-to-windows