Category Archives: Azure

AKS Private Cluster, kubenet, UDR, and Azure Firewall

This configuration will be covered the following requirement:

  • AKS Private Cluster with kubenet
  • Bring your own subnet with route table
  • limit egress with Azure Firewall
  • Access to ACR via public endpoint
  • Todo:
    • Access to ACR via private endpoint
    • Access to Azure Database for PostgreSQL via private endpoint
    • Monitoring via private link
    • Traffic flow query
    • Adding Azure Front Door/Application Gateway for public access

Private-AKS-AFW-egress-UDR
Private-AKS-AFW-egress-UDR

Creating AKS Private Cluster with Azure CLI

Sample output

a@myVM001:~$ ip address | grep "inet 10.42"
    inet 10.42.3.4/24 metric 100 brd 10.42.3.255 scope global eth0
a@myVM001:~$ kubectl get pods,svc
NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-676579fdc-68b6d   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-7kdwg   1/1     Running   0          6m30s
pod/nginx-deployment-676579fdc-q7q9f   1/1     Running   0          6m30s

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/fuju-nginx   LoadBalancer   10.0.141.238   10.42.1.7     80:31373/TCP   6m30s
service/kubernetes   ClusterIP      10.0.0.1       <none>        443/TCP        63m
a@myVM001:~$ curl -s http://10.42.1.7/ | grep title
<title>Welcome to nginx!</title>
a@myVM001:~$

Related Error Messages:

The following is “Error messages” when AKS Node unable to access ACR

a@myVM001:~$ az aks check-acr --resource-group $RG --name $AKSNAME --acr $MYACR.azurecr.io
Merged "aks-egress" as current context in /tmp/tmpaacrqofd
WARNING: version difference between client (1.26) and server (1.23) exceeds the supported minor version skew of +/-1
[2022-12-11T09:27:52Z] Checking host name resolution ($MYACR.azurecr.io): SUCCEEDED
[2022-12-11T09:27:52Z] Canonical name for ACR ($MYACR.azurecr.io): r0000ea.eastasia.cloudapp.azure.com.
[2022-12-11T09:27:52Z] ACR location: eastasia
[2022-12-11T09:27:52Z] Checking managed identity...
[2022-12-11T09:27:52Z] Kubelet managed identity client ID: 000-0000-0000-0000-0000c
[2022-12-11T09:27:53Z] Validating managed identity existance: SUCCEEDED
[2022-12-11T09:27:53Z] Validating image pull permission: FAILED
[2022-12-11T09:27:53Z] ACR myacregistry001.azurecr.io rejected token exchange: failed to send token exchange request: Post "https://$MYACR.azurecr.io/oauth2/exchange": EOF

The following is “Error message” when creating AKS cluster but $IDENTITY_ID doesn’t have permission to create entry in route table (UDR) – Additional information for built-in-roles (Network Contributor) can be found @ https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#network-contributor. And ” Microsoft.Network/routeTables/*” provider operations can be found details under https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftnetwork

 (CustomRouteTableMissingPermission) Managed identity or service principle must be given permission to read and write to custom route table /subscriptions/0000-0000-0000-0000-300a/resourceGroups/101-aks-egress-rg/providers/Microsoft.Network/routeTables/aks-egress-fwrt. Please see https://aka.ms/aks/customrt for more information

Sample Route Table after adding AKS ( Kubenet ) in VNET

Sample Route Table

See Also:

AKS – Private Cluster and PostgreSQL

Sample CLI:
* Creating ” AKS – Private Cluster with v-net peering ”
https://github.com/fujute/m18h/blob/master/aks/private-aks-with-vnet-peering.sh

* Creating AKS with ” Bring your own subnet and route table with kubenet”
https://github.com/fujute/m18h/blob/master/aks/private-aks-byo-subnet.sh

AKS  - Private Cluster - Bring your own subnet and route table with kubenet
AKS – Private Cluster – Bring your own subnet and route table with kubenet

See Also

Azure Application Insights customEvents and .NET 6

Getting customEvents with Azure Application Insights customEvents and .NET 6

dotnet new mvc -o videowebapp
cd videowebapp
dotnet add package Microsoft.ApplicationInsights.AspNetCore

Program.cs: Adding builder.Services.AddApplicationInsightsTelemetry();

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
IServiceCollection serviceCollection = builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddControllersWithViews();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}");

app.Run();

Getting InstrumentationKey

az resource show \
    --resource-group <resource_group_name> \
    --name <resource_name> \
    --resource-type "Microsoft.Insights/components" \
    --query properties.InstrumentationKey

appsettings.json : adding InstrumentationKey

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ApplicationInsights": {
    "InstrumentationKey": "0000000000000000000000000000000",
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Error"
    }
  }
}

HomeController.cs : Adding ” this.aiClient.TrackEvent(“CommentSubmitted”); ”

using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using videowebapp.Models;
using Microsoft.ApplicationInsights;

namespace videowebapp.Controllers;

public class HomeController : Controller
{
    private TelemetryClient aiClient;
 //   private readonly ILogger<HomeController> _logger;

public HomeController(TelemetryClient aiClient)
{
    this.aiClient = aiClient;
}
/*    public HomeController(ILogger<HomeController> logger)
    {
        _logger = logger;
    }
*/
    public IActionResult Index()
    {
       // _logger.LogWarning("fujuTE: An example of a Warning trace..");
       // _logger.LogError("fujuTE: An example of an Error level message");
        return View();
    }

    public IActionResult Privacy()
    { 
        // _logger.LogInformation("fujuTE: An example of a Information trace..");
        // Track an event
        this.aiClient.TrackEvent("CommentSubmitted");

        // Track an event with properties
        this.aiClient.TrackEvent("VideoUploaded", new Dictionary<string, string> {{"Category", "Sports"}, {"Format", "mp4"}});

        return View();
    }

    [ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)]
    public IActionResult Error()
    {
        return View(new ErrorViewModel { RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier });
    }
}
Application Insight – Transaction Search
Getting customEvents vai Log Analytics Workspace

See Also:

Terraform: Create Virtual Network Peering and VMs

Sample “Virtual network peering” with 2 VMs with terraform deployment

TF file: https://github.com/fujute/m18h/tree/master/tf/virtual-network-peering

  • main.tf
  • variables.tf

Building VNET peering with terraform

terraform plan -out main-vnet.tfplan
terraform apply "main-vnet.tfplan"

Sample screenshot to access fx1-vm1 in fx1-network1 via jump host fx1-vm2

ssh azureuser@10.0.2.4

adminuser@fx1-vm2:~$ hostname
fx1-vm2
adminuser@fx1-vm2:~$ ssh azureuser@10.0.2.4
azureuser@10.0.2.4's password:
Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 5.4.0-1080-azure x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jun  2 10:38:38 UTC 2022

  System load:  0.33              Processes:           131
  Usage of /:   6.5% of 28.90GB   Users logged in:     0
  Memory usage: 3%                IP address for eth0: 10.0.2.4
  Swap usage:   0%


0 updates can be applied immediately.

New release '20.04.4 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


Last login: Thu Jun  2 10:37:25 2022 from 192.168.2.4
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

azureuser@fx1-vm1:~$

delete the deployment with terraform destroy

terraform plan -destroy -out main-vnet.destroy.tfplan
terraform apply main-vnet.destroy.tfplan

Optional Tasks:

  • Adding Private Endpoint for Azure Blob Storage and Private DNS Zone

Reference command:

az vm image list-skus --location eastasia --offer WindowsServer --publisher MicrosoftWindowsServer
az vm image list-skus --location eastasia --offer UbuntuServer --publisher Canonical
az vm list-skus -l southeastasia  --resource-type virtualMachines  --output table | grep Standard_D2ds_v4

az account set --subscription "xxxxxxxxxxxxxxx"
az vm list-usage --location southeastasia -o table | grep -E -w -i  'DSv4|FSv2|ESv4'
#!/bin/bash
declare -a subscrptionsID=(
"12345-12342134-12342134-1234"
"12345-12342134-12342134-1235"
"12345-12342134-12342134-1236"
"12345-12342134-12342134-1237"
)

echo "${subscrptionsID[@]}"

for mySubscrptionsID in "${subscrptionsID[@]}" 
do
   az account set --subscription  “$mySubscrptionsID”
   az vm list-usage --location southeastasia -o table | grep -E -w -i  'DSv4|FSv2|ESv4|DSv3'
done

Deploy an ASP.NET Core and Azure SQL Database app to Azure App Service with Bicep,CLI and VS Code

Deploy software infrastructure with Bicep ( more details can be found from https://www.fuju.org/?p=38184

// based on https://www.fuju.org/?p=38184
RESOURCE_GROUP_NAME=23ExampleRG=23ExampleRG
az group create --name $RESOURCE_GROUP_NAME=23ExampleRG --location "eastasia"
az deployment group create --resource-group $RESOURCE_GROUP_NAME=23ExampleRG --template-file 01-main-web-app-sqldb.bicep  --parameters sqlAdministratorLogin='azuresqladmin' sqlAdministratorLoginPassword='MYSECRTEPASSWORD' location='eastasia'

Deploy sample ASP.NET Core Application to App Service with VS code

#https://www.fuju.org/?p=38044 
# https://docs.microsoft.com/en-us/azure/app-service/tutorial-dotnetcore-sqldb-app

git clone https://github.com/Azure-Samples/msdocs-app-service-sqldb-dotnetcore.git
cd msdocs-app-service-sqldb-dotnetcore
dotnet build .
dotnet publish -c Release
#Right-click on the generated publish folder in the Visual Studio Code explorer and #select Deploy to Web App.

Update database connection string and create database table with entity framework core

APP_SERVICE_NAME=websiteacoss37fw5new
RESOURCE_GROUP_NAME=23ExampleRG
SQL_SERVER_NAME=sqlserverRANDOM

az sql db show-connection-string \
    --client ado.net \
    --name sampledb \
    --server $SQL_SERVER_NAME.database.windows.net
	
az webapp config connection-string set \
    -g $RESOURCE_GROUP_NAME \
    -n $APP_SERVICE_NAME \
    -t SQLServer \
    --settings MyDbConnection="Server=tcp:$SQL_SERVER_NAME.database.windows.net,1433;Database=sampledb;User ID=azuresqladmin;Password='MYSECRETPASSWORD';Encrypt=true;Connection Timeout=30;"

# az sql server firewall-rule create --resource-group $RESOURCE_GROUP_NAME --server $SQL_SERVER_NAME.database.windows.net.database.windows.net --name LocalAccess --start-ip-address <your-ip> --end-ip-address <your-ip>

# Change database connection string in "appsettings.json" 
#"ConnectionStrings": {
#    "MyDbConnection": "Server=tcp:$SQL_SERVER_NAME.database.windows.net,1433;
#        Initial Catalog=sampledb;
#        Persist Security Info=False;
#        User ID=azuresqladmin;Password=MYSECRETPASSWORD;
#        Encrypt=True;
#        TrustServerCertificate=False;"
# }
  
# create database table in target database
dotnet tool install -g dotnet-ef
dotnet ef migrations add InitialCreat --project DotNetCoreSqlDb
dotnet ef database update --project DotNetCoreSqlDb

Access web app logs and delete resources group

az webapp log config \
    --web-server-logging 'filesystem' \
    --name $APP_SERVICE_NAME \
    --resource-group $RESOURCE_GROUP_NAME
	
	
az webapp log tail \
    --name $APP_SERVICE_NAME \
    --resource-group $RESOURCE_GROUP_NAME

az group delete --name $RESOURCE_GROUP_NAME

Azure App Service and Bicep

List of sample deployments of Azure App Service with various scenario by using Bicep

  • Azure App service with Blob
  • Azure App Service + SQLDB + App Insight
  • Azure App Service (Webapp + private endpoint (WebApp and SQL Server) + VNET integration.
  • Application Gateway (private and public listeners) -> Azure App Service (via PE) -> SQL Server (via PE)

Azure App service with Blob:
https://github.com/fujute/m18h/blob/master/bicep/00-main.bicep

# https://github.com/fujute/m18h/blob/master/bicep/00-main.bicep
RESOURCE_GROUP='21ExampleRG'
az group create --name $RESOURCE_GROUP--location westus3
az deployment group create --resource-group $RESOURCE_GROUP --template-file main.bicep --parameters environmentType=nonprod

az deployment group list --output table

Azure App Service + SQLDB + App Insight :
https://github.com/fujute/m18h/blob/master/bicep/01-main-web-app-sqldb.bicep

# https://github.com/fujute/m18h/blob/master/bicep/01-main-web-app-sqldb.bicep
RESOURCE_GROUP='22ExampleRG'
az group create --name $RESOURCE_GROUP --location westus3
az deployment group create --resource-group $RESOURCE_GROUP --template-file 01-main-web-app-sqldb.bicep  --parameters sqlAdministratorLogin='azuresqladmin' sqlAdministratorLoginPassword='PLEASECHANGEtoSECRETPASSWORD' location='westus3'

Azure App Service (Webapp + private endpoint (WebApp and SQL Server) + VNET integration:
Deploys two web apps (frontend and backend) and SQL Server securely connected together with VNet integration (webAppFrontend) and Private Endpoint(webAppBackend,SQLServer)
m18h/02-main-webapp-pe-vnet-integration.bicep at master · fujute/m18h (github.com)

# https://github.com/fujute/m18h/blob/master/bicep/02-main-webapp-pe-vnet-integration.bicep
RESOURCE_GROUP='23ExampleRG'
az group create --name $RESOURCE_GROUP--location westus3
az deployment group create --resource-group $RESOURCE_GROUP--template-file 02-main-webapp-pe-vnet-integration.bicep \
--parameters virtualNetworkName='m6290cvnet' sqlAdministratorLoginPassword='PLEASECHANGETOSECUREPASSWORD'

Reference

Integrate your app with an Azure virtual network

ref:
* https://docs.microsoft.com/en-us/azure/app-service/networking-features
* https://docs.microsoft.com/en-us/azure/app-service/overview-vnet-integration

picture: Regional virtual network integration

Application Gateway integration: Integration with App Service (multi-tenant)

Continue reading Azure App Service and Bicep

DevOps: Azure DevOps Release Gate (Query work items, SonarQube, Azure monitor)

A Sample of “Azure DevOps Release Gate ” the Gates with the following items

  • Query work items: Query Active Bug
  • SonarQube: ” Invoke REST API: POST”
  • Azure monitor: Azure monitor Alert

Software infra:

Based on ” Controlling Deployments using Release Gates | Azure DevOps Hands-on-Labs (azuredevopslabs.com) ” and ” Managing technical debt with SonarQube and Azure DevOps | Azure DevOps Hands-on-Labs (azuredevopslabs.com)

  • 2 Web app for sample canary and production
  • sonarqube in Azure Container Instance
AprRG="1TL-MyResourceGroup"
RNUMBER="041822"

az group create -n $AprRG -l eastasia
az appservice plan create -g $AprRG -n MyPlan --sku S1
az webapp create -g $AprRG -p MyPlan -n "PartsUnlimited-$RNUMBER-Canary"
az webapp create -g $AprRG -p MyPlan -n "PartsUnlimited-$RNUMBER-Prod"

RG_ID=$(az group create --name $AprRG  --location "eastasia" --query "id" --output tsv)
SERVICE_PRINCIPAL_NAME="Exzilla-sp-$RNUMBER"
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role contributor --scopes $RG_ID --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

az container create -g $AprRG --name sonarqubeaci180422 --image sonarqube --ports 9000 --dns-name-label mysonarqube200422 --cpu 2 --memory 3.5

#curl -u ea---fe: http://sonarqubeaci180422.exzilla.com:9000/api/qualitygates/project_status?projectKey=MyShuttle

Hint: Azure DevOps Release Gate with Azure DevOps Starter

To build quick demo for “Release Gate” with condition from Azure Board, Azure Monitor and SonarQube

  1. DevOps Starter ” .NET Core -> App Service ”
  2. Add SonarQube in “build” Pipeline
  3. Add Release ” UAT” Stage (Then, we have Dev & UAT)
  4. Add “Pre-deployment approvals”
  5. Add “Pre-deployment Gates” ” Query work items”
    Azure DevOps -> Boards -> Queries -> Active Bugs -> … -> Security -> ReleaseGate Build Service(myOrg) -> Read ( Allow )
  6. Add “monitoring” gate in “Dev” Stage
  7. Add Agentless Job in Tasks( manual Intervention )
  8. Add ” Post-deployment approvals” Gate “Query Azure Monitor Alerts”
  9. Add ” Post-deployment approvals” Gate “Invoke REST API:POST”

#URL suffix:
api/qualitygates/project_status?projectKey=MyShuttle

#success critirial:
eq(root['projectStatus'].status,'OK')

Sample of Canary releases (ref: What are deployment patterns? – Learn | Microsoft Docs )

DevTest and DevOps for microservice solutions

https://learn.microsoft.com/en-us/azure/architecture/solution-ideas/articles/dev-test-microservice

Labs:

See Also:

A user delegation SAS for blob with the Azure CLI & azcopy

Sample of using A user delegation SAS for blob with the Azure CLI and copy the blob file via azcopy with the SAS token

MY_RG=11-11RG
SUB_ID=<SUBID>
MY_SCOPES="/subscriptions/$SUB_ID/resourceGroups/$MY_RG/providers/Microsoft.Storage/storageAccounts/m14storage"
RG_ID=$(az group create --name $MY_RG  --location southeastasia --query "id" --output tsv)
SERVICE_PRINCIPAL_NAME=Exzilla-sp-14032022
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role "Storage Blob Data Contributor" --scopes $MY_SCOPES  --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

az login --service-principal -u $USER_NAME -p $PASSWORD --tenant <mytenant>.onmicrosoft.com

END=$(date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ')
SAS4BLOB=$(az storage blob generate-sas \
    --account-name m14storage \
    --container-name data0314 \
    --name "kblob-file-001.txt" \
    --permissions acdrw \
    --expiry $END \
    --auth-mode login \
    --as-user \
    --full-uri )
	
azcopy copy $SAS4BLOB  . 

See Also:

Questions:

  • SP’s Password protection ?

Create Azure Service Principal and “az login”

Sample of creating Resource Group and then create Service Principle with contributor role under the RG for DevOps Pipeline

#MY_RG=1myResourceGroup
#az group create --name $MY_RG --location southeastasia
#RG_ID=$(az group show --name $MY_RG --query "id" --output tsv)
# or 
MY_RG=1myResourceGroup
RG_ID=$(az group create --name $MY_RG  --location southeastasia --query "id" --output tsv)
SERVICE_PRINCIPAL_NAME=Exzilla-sp-21022022
PASSWORD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role contributor --scopes $RG_ID  --query "password" --output tsv)
USER_NAME=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query "[].appId" --output tsv)

az login --service-principal -u $USER_NAME -p $PASSWORD --tenant <mytenant>.onmicrosoft.com

Connect to SQL Database from App Service by using a managed identity – System-assigned.

Running App Service with Azure SQL based

Fix list:
The steps in this guide has been tested with “.NET 6.0”

dotnet tool install -g dotnet-ef
dotnet ef migrations add InitialCreate
dotnet ef database update
dotnet add package Microsoft.Data.SqlClient --version 4.0.1
dotnet add package Azure.Identity --version 1.5.0

appsettings.json :

{
  "Logging": {
    "LogLevel": {
      "Default": "Information"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "CoredBConnection": "Server=tcp:mydatabase-server-sqlsrv.database.windows.net;Authentication=Active Directory Default; Database=CoredB;"
  }
} 
az webapp identity assign --resource-group my-002-rg --name webapp-core-sql-018
CREATE USER [webapp-core-sql-018] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [webapp-core-sql-018];
ALTER ROLE db_datawriter ADD MEMBER [webapp-core-sql-018];
ALTER ROLE db_ddladmin ADD MEMBER [webapp-core-sql-018];
GO

Program.cs :

public void ConfigureServices(IServiceCollection services)
{
   services.AddControllersWithViews();
   services.AddDbContext<MyDatabaseContext>(options =>
            options.UseSqlServer(Configuration.GetConnectionString("CoredBConnection")));
}

See Also:

Web App & Azure SQL Templates:

  • https://docs.microsoft.com/en-us/azure/azure-sql/database/arm-templates-content-guide?tabs=single-database
  • https://docs.microsoft.com/en-us/azure/app-service/samples-resource-manager-templates
  • https://azure.microsoft.com/en-au/resources/templates/web-app-sql-database/