AKS Private Cluster, kubenet, UDR, and Azure Firewall

This configuration will be covered the following requirement:

  • AKS Private Cluster with kubenet
  • Bring your own subnet with route table
  • limit egress with Azure Firewall
  • Access to ACR via public endpoint
  • Todo:
    • Access to ACR via private endpoint
    • Access to Azure Database for PostgreSQL via private endpoint
    • Monitoring via private link
    • Traffic flow query
    • Adding Azure Front Door/Application Gateway for public access

Private-AKS-AFW-egress-UDR
Private-AKS-AFW-egress-UDR

Creating AKS Private Cluster with Azure CLI

Sample output

a@myVM001:~$ ip address | grep "inet 10.42"
    inet 10.42.3.4/24 metric 100 brd 10.42.3.255 scope global eth0
a@myVM001:~$ kubectl get pods,svc
NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-676579fdc-68b6d   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-7kdwg   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-q7q9f   1/1     Running   0          6m30s

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/fuju-nginx   LoadBalancer   10.0.141.238   10.42.1.7     80:31373/TCP   6m30s
service/kubernetes   ClusterIP      10.0.0.1       <none>        443/TCP        63m
a@myVM001:~$ curl -s http://10.42.1.7/ | grep title
<title>Welcome to nginx!</title>
a@myVM001:~$

Related Error Messages:

The following is “Error messages” when AKS Node unable to access ACR

a@myVM001:~$ az aks check-acr --resource-group $RG --name $AKSNAME --acr $MYACR.azurecr.io
Merged "aks-egress" as current context in /tmp/tmpaacrqofd
WARNING: version difference between client (1.26) and server (1.23) exceeds the supported minor version skew of +/-1
[2022-12-11T09:27:52Z] Checking host name resolution ($MYACR.azurecr.io): SUCCEEDED
[2022-12-11T09:27:52Z] Canonical name for ACR ($MYACR.azurecr.io): r0000ea.eastasia.cloudapp.azure.com.
[2022-12-11T09:27:52Z] ACR location: eastasia
[2022-12-11T09:27:52Z] Checking managed identity...
[2022-12-11T09:27:52Z] Kubelet managed identity client ID: 000-0000-0000-0000-0000c
[2022-12-11T09:27:53Z] Validating managed identity existance: SUCCEEDED
[2022-12-11T09:27:53Z] Validating image pull permission: FAILED
[2022-12-11T09:27:53Z] ACR myacregistry001.azurecr.io rejected token exchange: failed to send token exchange request: Post "https://$MYACR.azurecr.io/oauth2/exchange": EOF

The following is “Error message” when creating AKS cluster but $IDENTITY_ID doesn’t have permission to create entry in route table (UDR) – Additional information for built-in-roles (Network Contributor) can be found @ https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#network-contributor. And ” Microsoft.Network/routeTables/*” provider operations can be found details under https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftnetwork

 (CustomRouteTableMissingPermission) Managed identity or service principle must be given permission to read and write to custom route table /subscriptions/0000-0000-0000-0000-300a/resourceGroups/101-aks-egress-rg/providers/Microsoft.Network/routeTables/aks-egress-fwrt. Please see https://aka.ms/aks/customrt for more information

Sample Route Table after adding AKS ( Kubenet ) in VNET

Sample Route Table

See Also:

AKS – Private Cluster and PostgreSQL

Sample CLI:
* Creating ” AKS – Private Cluster with v-net peering ”
https://github.com/fujute/m18h/blob/master/aks/private-aks-with-vnet-peering.sh

* Creating AKS with ” Bring your own subnet and route table with kubenet”
https://github.com/fujute/m18h/blob/master/aks/private-aks-byo-subnet.sh

AKS  - Private Cluster - Bring your own subnet and route table with kubenet
AKS – Private Cluster – Bring your own subnet and route table with kubenet

See Also

Azure Application Insights customEvents and .NET 6

Getting customEvents with Azure Application Insights customEvents and .NET 6

dotnet new mvc -o videowebapp
cd videowebapp
dotnet add package Microsoft.ApplicationInsights.AspNetCore

Program.cs: Adding builder.Services.AddApplicationInsightsTelemetry();

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
IServiceCollection serviceCollection = builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddControllersWithViews();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}");

app.Run();

Getting InstrumentationKey

az resource show \
    --resource-group <resource_group_name> \
    --name <resource_name> \
    --resource-type "Microsoft.Insights/components" \
    --query properties.InstrumentationKey

appsettings.json : adding InstrumentationKey

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ApplicationInsights": {
    "InstrumentationKey": "0000000000000000000000000000000",
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Error"
    }
  }
}

HomeController.cs : Adding ” this.aiClient.TrackEvent(“CommentSubmitted”); ”

using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using videowebapp.Models;
using Microsoft.ApplicationInsights;

namespace videowebapp.Controllers;

public class HomeController : Controller
{
    private TelemetryClient aiClient;
 //   private readonly ILogger<HomeController> _logger;

public HomeController(TelemetryClient aiClient)
{
    this.aiClient = aiClient;
}
/*    public HomeController(ILogger<HomeController> logger)
    {
        _logger = logger;
    }
*/
    public IActionResult Index()
    {
       // _logger.LogWarning("fujuTE: An example of a Warning trace..");
       // _logger.LogError("fujuTE: An example of an Error level message");
        return View();
    }

    public IActionResult Privacy()
    { 
        // _logger.LogInformation("fujuTE: An example of a Information trace..");
        // Track an event
        this.aiClient.TrackEvent("CommentSubmitted");

        // Track an event with properties
        this.aiClient.TrackEvent("VideoUploaded", new Dictionary<string, string> {{"Category", "Sports"}, {"Format", "mp4"}});

        return View();
    }

    [ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)]
    public IActionResult Error()
    {
        return View(new ErrorViewModel { RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier });
    }
}
Application Insight – Transaction Search
Getting customEvents vai Log Analytics Workspace

See Also:

Terraform: Create Virtual Network Peering and VMs

Sample “Virtual network peering” with 2 VMs with terraform deployment

TF file: https://github.com/fujute/m18h/tree/master/tf/virtual-network-peering

  • main.tf
  • variables.tf

Building VNET peering with terraform

terraform plan -out main-vnet.tfplan
terraform apply "main-vnet.tfplan"

Sample screenshot to access fx1-vm1 in fx1-network1 via jump host fx1-vm2

ssh azureuser@10.0.2.4

adminuser@fx1-vm2:~$ hostname
fx1-vm2
adminuser@fx1-vm2:~$ ssh azureuser@10.0.2.4
azureuser@10.0.2.4's password:
Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 5.4.0-1080-azure x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jun  2 10:38:38 UTC 2022

  System load:  0.33              Processes:           131
  Usage of /:   6.5% of 28.90GB   Users logged in:     0
  Memory usage: 3%                IP address for eth0: 10.0.2.4
  Swap usage:   0%


0 updates can be applied immediately.

New release '20.04.4 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


Last login: Thu Jun  2 10:37:25 2022 from 192.168.2.4
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

azureuser@fx1-vm1:~$

delete the deployment with terraform destroy

terraform plan -destroy -out main-vnet.destroy.tfplan
terraform apply main-vnet.destroy.tfplan

Optional Tasks:

  • Adding Private Endpoint for Azure Blob Storage and Private DNS Zone

Reference command:

az vm image list-skus --location eastasia --offer WindowsServer --publisher MicrosoftWindowsServer
az vm image list-skus --location eastasia --offer UbuntuServer --publisher Canonical
az vm list-skus -l southeastasia  --resource-type virtualMachines  --output table | grep Standard_D2ds_v4

az account set --subscription "xxxxxxxxxxxxxxx"
az vm list-usage --location southeastasia -o table | grep -E -w -i  'DSv4|FSv2|ESv4'
#!/bin/bash
declare -a subscrptionsID=(
"12345-12342134-12342134-1234"
"12345-12342134-12342134-1235"
"12345-12342134-12342134-1236"
"12345-12342134-12342134-1237"
)

echo "${subscrptionsID[@]}"

for mySubscrptionsID in "${subscrptionsID[@]}" 
do
   az account set --subscription  “$mySubscrptionsID”
   az vm list-usage --location southeastasia -o table | grep -E -w -i  'DSv4|FSv2|ESv4|DSv3'
done

Deploy an ASP.NET Core and Azure SQL Database app to Azure App Service with Bicep,CLI and VS Code

Deploy software infrastructure with Bicep ( more details can be found from https://www.fuju.org/?p=38184

// based on https://www.fuju.org/?p=38184
RESOURCE_GROUP_NAME=23ExampleRG=23ExampleRG
az group create --name $RESOURCE_GROUP_NAME=23ExampleRG --location "eastasia"
az deployment group create --resource-group $RESOURCE_GROUP_NAME=23ExampleRG --template-file 01-main-web-app-sqldb.bicep  --parameters sqlAdministratorLogin='azuresqladmin' sqlAdministratorLoginPassword='MYSECRTEPASSWORD' location='eastasia'

Deploy sample ASP.NET Core Application to App Service with VS code

#https://www.fuju.org/?p=38044 
# https://docs.microsoft.com/en-us/azure/app-service/tutorial-dotnetcore-sqldb-app

git clone https://github.com/Azure-Samples/msdocs-app-service-sqldb-dotnetcore.git
cd msdocs-app-service-sqldb-dotnetcore
dotnet build .
dotnet publish -c Release
#Right-click on the generated publish folder in the Visual Studio Code explorer and #select Deploy to Web App.

Update database connection string and create database table with entity framework core

APP_SERVICE_NAME=websiteacoss37fw5new
RESOURCE_GROUP_NAME=23ExampleRG
SQL_SERVER_NAME=sqlserverRANDOM

az sql db show-connection-string \
    --client ado.net \
    --name sampledb \
    --server $SQL_SERVER_NAME.database.windows.net
	
az webapp config connection-string set \
    -g $RESOURCE_GROUP_NAME \
    -n $APP_SERVICE_NAME \
    -t SQLServer \
    --settings MyDbConnection="Server=tcp:$SQL_SERVER_NAME.database.windows.net,1433;Database=sampledb;User ID=azuresqladmin;Password='MYSECRETPASSWORD';Encrypt=true;Connection Timeout=30;"

# az sql server firewall-rule create --resource-group $RESOURCE_GROUP_NAME --server $SQL_SERVER_NAME.database.windows.net.database.windows.net --name LocalAccess --start-ip-address <your-ip> --end-ip-address <your-ip>

# Change database connection string in "appsettings.json" 
#"ConnectionStrings": {
#    "MyDbConnection": "Server=tcp:$SQL_SERVER_NAME.database.windows.net,1433;
#        Initial Catalog=sampledb;
#        Persist Security Info=False;
#        User ID=azuresqladmin;Password=MYSECRETPASSWORD;
#        Encrypt=True;
#        TrustServerCertificate=False;"
# }
  
# create database table in target database
dotnet tool install -g dotnet-ef
dotnet ef migrations add InitialCreat --project DotNetCoreSqlDb
dotnet ef database update --project DotNetCoreSqlDb

Access web app logs and delete resources group

az webapp log config \
    --web-server-logging 'filesystem' \
    --name $APP_SERVICE_NAME \
    --resource-group $RESOURCE_GROUP_NAME
	
	
az webapp log tail \
    --name $APP_SERVICE_NAME \
    --resource-group $RESOURCE_GROUP_NAME

az group delete --name $RESOURCE_GROUP_NAME

Azure App Service and Bicep

List of sample deployments of Azure App Service with various scenario by using Bicep

  • Azure App service with Blob
  • Azure App Service + SQLDB + App Insight
  • Azure App Service (Webapp + private endpoint (WebApp and SQL Server) + VNET integration.
  • Application Gateway (private and public listeners) -> Azure App Service (via PE) -> SQL Server (via PE)

Azure App service with Blob:
https://github.com/fujute/m18h/blob/master/bicep/00-main.bicep

# https://github.com/fujute/m18h/blob/master/bicep/00-main.bicep
RESOURCE_GROUP='21ExampleRG'
az group create --name $RESOURCE_GROUP--location westus3
az deployment group create --resource-group $RESOURCE_GROUP --template-file main.bicep --parameters environmentType=nonprod

az deployment group list --output table

Azure App Service + SQLDB + App Insight :
https://github.com/fujute/m18h/blob/master/bicep/01-main-web-app-sqldb.bicep

# https://github.com/fujute/m18h/blob/master/bicep/01-main-web-app-sqldb.bicep
RESOURCE_GROUP='22ExampleRG'
az group create --name $RESOURCE_GROUP --location westus3
az deployment group create --resource-group $RESOURCE_GROUP --template-file 01-main-web-app-sqldb.bicep  --parameters sqlAdministratorLogin='azuresqladmin' sqlAdministratorLoginPassword='PLEASECHANGEtoSECRETPASSWORD' location='westus3'

Azure App Service (Webapp + private endpoint (WebApp and SQL Server) + VNET integration:
Deploys two web apps (frontend and backend) and SQL Server securely connected together with VNet integration (webAppFrontend) and Private Endpoint(webAppBackend,SQLServer)
m18h/02-main-webapp-pe-vnet-integration.bicep at master · fujute/m18h (github.com)

# https://github.com/fujute/m18h/blob/master/bicep/02-main-webapp-pe-vnet-integration.bicep
RESOURCE_GROUP='23ExampleRG'
az group create --name $RESOURCE_GROUP--location westus3
az deployment group create --resource-group $RESOURCE_GROUP--template-file 02-main-webapp-pe-vnet-integration.bicep \
--parameters virtualNetworkName='m6290cvnet' sqlAdministratorLoginPassword='PLEASECHANGETOSECUREPASSWORD'

Reference

Integrate your app with an Azure virtual network

ref:
* https://docs.microsoft.com/en-us/azure/app-service/networking-features
* https://docs.microsoft.com/en-us/azure/app-service/overview-vnet-integration

picture: Regional virtual network integration

Application Gateway integration: Integration with App Service (multi-tenant)

Continue reading Azure App Service and Bicep

Dynamically create and use a persistent volume with Azure Files in Azure Kubernetes Service (AKS)

How to dynamically create an Azure Files share for use by multiple pods in an Azure Kubernetes Service (AKS) cluster

This sample has been tested with ” AKS, Azure File, PV,PVC, Deployment, Multi-Container ”

Sample file can be accessed from at “https://github.com/fujute/apr5

az aks get-credentials --admin --name MyManagedCluster --resource-group MyResourceGroup
kubectl apply -f storageclass-and-pvc.yaml
kubectl apply -f fuju-nginx-azfile.yaml

kubectl get pod

kubectl exec fuju-nginx-azfile-5c55fddc9f-6tgk  -c 1st-c -- /bin/cat /usr/share/nginx/html/index2.html
kubectl exec fuju-nginx-azfile-5c55fddc9f-6tgk6 -c 1st-c -- /bin/cat /mnt/config-1st/my-config-file.txt
kubectl exec fuju-nginx-azfile-5c55fddc9f-6tgk6 -c 2nd-c -- /bin/cat /mnt/config-2nd/my-config-file.txt

DevOps: Azure DevOps Release Gate (Query work items, SonarQube, Azure monitor)

A Sample of “Azure DevOps Release Gate ” the Gates with the following items

  • Query work items: Query Active Bug
  • SonarQube: ” Invoke REST API: POST”
  • Azure monitor: Azure monitor Alert

Software infra:

Based on ” Controlling Deployments using Release Gates | Azure DevOps Hands-on-Labs (azuredevopslabs.com) ” and ” Managing technical debt with SonarQube and Azure DevOps | Azure DevOps Hands-on-Labs (azuredevopslabs.com)

  • 2 Web app for sample canary and production
  • sonarqube in Azure Container Instance
AprRG="1TL-MyResourceGroup"
RNUMBER="041822"

az group create -n $AprRG -l eastasia
az appservice plan create -g $AprRG -n MyPlan --sku S1
az webapp create -g $AprRG -p MyPlan -n "PartsUnlimited-$RNUMBER-Canary"
az webapp create -g $AprRG -p MyPlan -n "PartsUnlimited-$RNUMBER-Prod"

RG_ID=$(az group create --name $AprRG  --location "eastasia" --query "id" --output tsv)
SERVICE_PRINCIPAL_NAME="Exzilla-sp-$RNUMBER"
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role contributor --scopes $RG_ID --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

az container create -g $AprRG --name sonarqubeaci180422 --image sonarqube --ports 9000 --dns-name-label mysonarqube200422 --cpu 2 --memory 3.5

#curl -u ea---fe: http://sonarqubeaci180422.exzilla.com:9000/api/qualitygates/project_status?projectKey=MyShuttle

Hint: Azure DevOps Release Gate with Azure DevOps Starter

To build quick demo for “Release Gate” with condition from Azure Board, Azure Monitor and SonarQube

  1. DevOps Starter ” .NET Core -> App Service ”
  2. Add SonarQube in “build” Pipeline
  3. Add Release ” UAT” Stage (Then, we have Dev & UAT)
  4. Add “Pre-deployment approvals”
  5. Add “Pre-deployment Gates” ” Query work items”
    Azure DevOps -> Boards -> Queries -> Active Bugs -> … -> Security -> ReleaseGate Build Service(myOrg) -> Read ( Allow )
  6. Add “monitoring” gate in “Dev” Stage
  7. Add Agentless Job in Tasks( manual Intervention )
  8. Add ” Post-deployment approvals” Gate “Query Azure Monitor Alerts”
  9. Add ” Post-deployment approvals” Gate “Invoke REST API:POST”

#URL suffix:
api/qualitygates/project_status?projectKey=MyShuttle

#success critirial:
eq(root['projectStatus'].status,'OK')

Sample of Canary releases (ref: What are deployment patterns? – Learn | Microsoft Docs )

DevTest and DevOps for microservice solutions

https://learn.microsoft.com/en-us/azure/architecture/solution-ideas/articles/dev-test-microservice

Labs:

See Also:

A user delegation SAS for blob with the Azure CLI & azcopy

Sample of using A user delegation SAS for blob with the Azure CLI and copy the blob file via azcopy with the SAS token

MY_RG=11-11RG
SUB_ID=<SUBID>
MY_SCOPES="/subscriptions/$SUB_ID/resourceGroups/$MY_RG/providers/Microsoft.Storage/storageAccounts/m14storage"
RG_ID=$(az group create --name $MY_RG  --location southeastasia --query "id" --output tsv)
SERVICE_PRINCIPAL_NAME=Exzilla-sp-14032022
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role "Storage Blob Data Contributor" --scopes $MY_SCOPES  --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

az login --service-principal -u $USER_NAME -p $PASSWORD --tenant <mytenant>.onmicrosoft.com

END=$(date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ')
SAS4BLOB=$(az storage blob generate-sas \
    --account-name m14storage \
    --container-name data0314 \
    --name "kblob-file-001.txt" \
    --permissions acdrw \
    --expiry $END \
    --auth-mode login \
    --as-user \
    --full-uri )
	
azcopy copy $SAS4BLOB  . 

See Also:

Questions:

  • SP’s Password protection ?

Create Azure Service Principal and “az login”

Sample of creating Resource Group and then create Service Principle with contributor role under the RG for DevOps Pipeline

#MY_RG=1myResourceGroup
#az group create --name $MY_RG --location southeastasia
#RG_ID=$(az group show --name $MY_RG --query "id" --output tsv)
# or 
MY_RG=1myResourceGroup
RG_ID=$(az group create --name $MY_RG  --location southeastasia --query "id" --output tsv)
SERVICE_PRINCIPAL_NAME=Exzilla-sp-21022022
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role contributor --scopes $RG_ID  --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

az login --service-principal -u $USER_NAME -p $PASSWORD --tenant <mytenant>.onmicrosoft.com