This configuration will be covered the following requirement:

• AKS Private Cluster with kubenet
• Bring your own subnet with route table
• limit egress with Azure Firewall
• Access to ACR via public endpoint
• Todo:
  • Access to ACR via private endpoint
  • Access to Azure Database for PostgreSQL via private endpoint
  • Monitoring via private link
  • Traffic flow query
  • Adding Azure Front Door/Application Gateway for public access

Creating AKS Private Cluster with Azure CLI

• https://github.com/fujute/m18h/blob/master/aks/private-aks-kubenet-w-firewall.sh

Sample output

a@myVM001:~$ ip address | grep "inet 10.42"
    inet 10.42.3.4/24 metric 100 brd 10.42.3.255 scope global eth0
a@myVM001:~$ kubectl get pods,svc
NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-676579fdc-68b6d   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-7kdwg   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-q7q9f   1/1     Running   0          6m30s

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/fuju-nginx   LoadBalancer   10.0.141.238   10.42.1.7     80:31373/TCP   6m30s
service/kubernetes   ClusterIP      10.0.0.1       <none>        443/TCP        63m
a@myVM001:~$ curl -s http://10.42.1.7/ | grep title
<title>Welcome to nginx!</title>
a@myVM001:~$

Related Error Messages:

The following is “Error messages” when AKS Node unable to access ACR

a@myVM001:~$ az aks check-acr --resource-group $RG --name $AKSNAME --acr $MYACR.azurecr.io
Merged "aks-egress" as current context in /tmp/tmpaacrqofd
WARNING: version difference between client (1.26) and server (1.23) exceeds the supported minor version skew of +/-1
[2022-12-11T09:27:52Z] Checking host name resolution ($MYACR.azurecr.io): SUCCEEDED
[2022-12-11T09:27:52Z] Canonical name for ACR ($MYACR.azurecr.io): r0000ea.eastasia.cloudapp.azure.com.
[2022-12-11T09:27:52Z] ACR location: eastasia
[2022-12-11T09:27:52Z] Checking managed identity...
[2022-12-11T09:27:52Z] Kubelet managed identity client ID: 000-0000-0000-0000-0000c
[2022-12-11T09:27:53Z] Validating managed identity existance: SUCCEEDED
[2022-12-11T09:27:53Z] Validating image pull permission: FAILED
[2022-12-11T09:27:53Z] ACR myacregistry001.azurecr.io rejected token exchange: failed to send token exchange request: Post "https://$MYACR.azurecr.io/oauth2/exchange": EOF

The following is “Error message” when creating AKS cluster but $IDENTITY_ID doesn’t have permission to create entry in route table (UDR) – Additional information for built-in-roles (Network Contributor) can be found @ https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#network-contributor. And ” Microsoft.Network/routeTables/*” provider operations can be found details under https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftnetwork ”

 (CustomRouteTableMissingPermission) Managed identity or service principle must be given permission to read and write to custom route table /subscriptions/0000-0000-0000-0000-300a/resourceGroups/101-aks-egress-rg/providers/Microsoft.Network/routeTables/aks-egress-fwrt. Please see https://aka.ms/aks/customrt for more information

Sample Route Table after adding AKS ( Kubenet ) in VNET

See Also:

• limit egress traffic
  • https://learn.microsoft.com/en-us/azure/aks/egress-outboundtype
  • https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic
  • https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic#restrict-egress-traffic-using-azure-firewall
• https://learn.microsoft.com/en-us/azure/aks/configure-kubenet#bring-your-own-subnet-and-route-table-with-kubenet
• CyberSecurity
  • Microsoft Cybersecurity Reference Architectures
• Intergration
  • https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity
  • https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/tutorial-django-aks-database
• Firewall related
  • https://learn.microsoft.com/en-us/azure/container-registry/container-registry-firewall-access-rules
  • https://github.com/microsoft/containerregistry/blob/main/client-firewall-rules.md
  • https://learn.microsoft.com/en-us/services-hub/health/assessments-troubleshooting
  • https://learn.microsoft.com/en-us/azure/architecture/example-scenario/aks-firewall/aks-firewall
  • https://www.microsoft.com/en-us/download/details.aspx?id=56519
    • {Workspace GUID}.ods.opinsights.azure.com
    • {Workspace GUID}.oms.opinsights.azure.com
• Choosing a network model to use
  • https://learn.microsoft.com/en-us/azure/aks/configure-kubenet
  • https://learn.microsoft.com/en-us/azure/aks/concepts-network
• Baseline
  • https://learn.microsoft.com/en-us/azure/architecture/reference-architectures/containers/aks/baseline-aks
  • https://learn.microsoft.com/en-us/azure/architecture/reference-architectures/containers/aks-pci/aks-pci-ra-code-assets
• IP Ranges
  • Azure IP Ranges and Service Tags – Public Cloud : https://www.microsoft.com/en-us/download/details.aspx?id=56519
  • https://learn.microsoft.com/en-us/azure/azure-portal/azure-portal-safelist-urls?tabs=public-cloud
  • https://learn.microsoft.com/en-us/answers/questions/421629/whitelisted-url-to-enable-user-sign-in-to-windows