Category Archives: Workshop

AKS Private Cluster, kubenet, UDR, and Azure Firewall

This configuration will be covered the following requirement:

  • AKS Private Cluster with kubenet
  • Bring your own subnet with route table
  • limit egress with Azure Firewall
  • Access to ACR via public endpoint
  • Todo:
    • Access to ACR via private endpoint
    • Access to Azure Database for PostgreSQL via private endpoint
    • Monitoring via private link
    • Traffic flow query
    • Adding Azure Front Door/Application Gateway for public access


Creating AKS Private Cluster with Azure CLI

Sample output

a@myVM001:~$ ip address | grep "inet 10.42"
    inet metric 100 brd scope global eth0
a@myVM001:~$ kubectl get pods,svc
NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-676579fdc-68b6d   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-7kdwg   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-q7q9f   1/1     Running   0          6m30s

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/fuju-nginx   LoadBalancer     80:31373/TCP   6m30s
service/kubernetes   ClusterIP       <none>        443/TCP        63m
a@myVM001:~$ curl -s | grep title
<title>Welcome to nginx!</title>

Related Error Messages:

The following is “Error messages” when AKS Node unable to access ACR

a@myVM001:~$ az aks check-acr --resource-group $RG --name $AKSNAME --acr $
Merged "aks-egress" as current context in /tmp/tmpaacrqofd
WARNING: version difference between client (1.26) and server (1.23) exceeds the supported minor version skew of +/-1
[2022-12-11T09:27:52Z] Checking host name resolution ($ SUCCEEDED
[2022-12-11T09:27:52Z] Canonical name for ACR ($
[2022-12-11T09:27:52Z] ACR location: eastasia
[2022-12-11T09:27:52Z] Checking managed identity...
[2022-12-11T09:27:52Z] Kubelet managed identity client ID: 000-0000-0000-0000-0000c
[2022-12-11T09:27:53Z] Validating managed identity existance: SUCCEEDED
[2022-12-11T09:27:53Z] Validating image pull permission: FAILED
[2022-12-11T09:27:53Z] ACR rejected token exchange: failed to send token exchange request: Post "https://$": EOF

The following is “Error message” when creating AKS cluster but $IDENTITY_ID doesn’t have permission to create entry in route table (UDR) – Additional information for built-in-roles (Network Contributor) can be found @ And ” Microsoft.Network/routeTables/*” provider operations can be found details under

 (CustomRouteTableMissingPermission) Managed identity or service principle must be given permission to read and write to custom route table /subscriptions/0000-0000-0000-0000-300a/resourceGroups/101-aks-egress-rg/providers/Microsoft.Network/routeTables/aks-egress-fwrt. Please see for more information

Sample Route Table after adding AKS ( Kubenet ) in VNET

Sample Route Table

See Also:

AKS – Private Cluster and PostgreSQL

Sample CLI:
* Creating ” AKS – Private Cluster with v-net peering ”

* Creating AKS with ” Bring your own subnet and route table with kubenet”

AKS  - Private Cluster - Bring your own subnet and route table with kubenet
AKS – Private Cluster – Bring your own subnet and route table with kubenet

See Also

Terraform: Create Virtual Network Peering and VMs

Sample “Virtual network peering” with 2 VMs with terraform deployment

TF file:


Building VNET peering with terraform

terraform plan -out main-vnet.tfplan
terraform apply "main-vnet.tfplan"

Sample screenshot to access fx1-vm1 in fx1-network1 via jump host fx1-vm2

ssh azureuser@

adminuser@fx1-vm2:~$ hostname
adminuser@fx1-vm2:~$ ssh azureuser@
azureuser@'s password:
Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 5.4.0-1080-azure x86_64)

 * Documentation:
 * Management:
 * Support:

  System information as of Thu Jun  2 10:38:38 UTC 2022

  System load:  0.33              Processes:           131
  Usage of /:   6.5% of 28.90GB   Users logged in:     0
  Memory usage: 3%                IP address for eth0:
  Swap usage:   0%

0 updates can be applied immediately.

New release '20.04.4 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Thu Jun  2 10:37:25 2022 from
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.


delete the deployment with terraform destroy

terraform plan -destroy -out main-vnet.destroy.tfplan
terraform apply main-vnet.destroy.tfplan

Optional Tasks:

  • Adding Private Endpoint for Azure Blob Storage and Private DNS Zone

Reference command:

az vm image list-skus --location eastasia --offer WindowsServer --publisher MicrosoftWindowsServer
az vm image list-skus --location eastasia --offer UbuntuServer --publisher Canonical
az vm list-skus -l southeastasia  --resource-type virtualMachines  --output table | grep Standard_D2ds_v4

az account set --subscription "xxxxxxxxxxxxxxx"
az vm list-usage --location southeastasia -o table | grep -E -w -i  'DSv4|FSv2|ESv4'
declare -a subscrptionsID=(

echo "${subscrptionsID[@]}"

for mySubscrptionsID in "${subscrptionsID[@]}" 
   az account set --subscription  “$mySubscrptionsID”
   az vm list-usage --location southeastasia -o table | grep -E -w -i  'DSv4|FSv2|ESv4|DSv3'

DevOps: Azure DevOps Release Gate (Query work items, SonarQube, Azure monitor)

A Sample of “Azure DevOps Release Gate ” the Gates with the following items

  • Query work items: Query Active Bug
  • SonarQube: ” Invoke REST API: POST”
  • Azure monitor: Azure monitor Alert

Software infra:

Based on ” Controlling Deployments using Release Gates | Azure DevOps Hands-on-Labs ( ” and ” Managing technical debt with SonarQube and Azure DevOps | Azure DevOps Hands-on-Labs (

  • 2 Web app for sample canary and production
  • sonarqube in Azure Container Instance

az group create -n $AprRG -l eastasia
az appservice plan create -g $AprRG -n MyPlan --sku S1
az webapp create -g $AprRG -p MyPlan -n "PartsUnlimited-$RNUMBER-Canary"
az webapp create -g $AprRG -p MyPlan -n "PartsUnlimited-$RNUMBER-Prod"

RG_ID=$(az group create --name $AprRG  --location "eastasia" --query "id" --output tsv)
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role contributor --scopes $RG_ID --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

az container create -g $AprRG --name sonarqubeaci180422 --image sonarqube --ports 9000 --dns-name-label mysonarqube200422 --cpu 2 --memory 3.5

#curl -u ea---fe:

Hint: Azure DevOps Release Gate with Azure DevOps Starter

To build quick demo for “Release Gate” with condition from Azure Board, Azure Monitor and SonarQube

  1. DevOps Starter ” .NET Core -> App Service ”
  2. Add SonarQube in “build” Pipeline
  3. Add Release ” UAT” Stage (Then, we have Dev & UAT)
  4. Add “Pre-deployment approvals”
  5. Add “Pre-deployment Gates” ” Query work items”
    Azure DevOps -> Boards -> Queries -> Active Bugs -> … -> Security -> ReleaseGate Build Service(myOrg) -> Read ( Allow )
  6. Add “monitoring” gate in “Dev” Stage
  7. Add Agentless Job in Tasks( manual Intervention )
  8. Add ” Post-deployment approvals” Gate “Query Azure Monitor Alerts”
  9. Add ” Post-deployment approvals” Gate “Invoke REST API:POST”

#URL suffix:

#success critirial:

Sample of Canary releases (ref: What are deployment patterns? – Learn | Microsoft Docs )

DevTest and DevOps for microservice solutions


See Also: