Adding a User Defined Route for API Management with a Service Tag via Bicep


azure

If you are setting up API Management with Internal network (vNet/Subnet) and a public IP.
There are a few prepreqs, among them is to add an NSG to the subnet with a number of predefined rules, these are documented by Microsoft and are rather easy to get hold of.
What also is a requirement, is if you have a peered vNet…then you need to add a Route table to the subnet with a User Defined Route, this to route the API Management traffic straight out to the internet to access the management service.

If you create the APIM service without the route, you will get an error that the APIM management service can’t be contected.

Running the diagnostic in the network gui gets us this dialogue:

APIMDiagnose

The UDR is easy enough to add using the portal, but…if you are a IaC shop, and want to deploy using Bicep, then its an undocumented feature.

And you do want to use the Service Tag, because the IP/IP range can change to the API Management.

I solved it after some research, this is what you do.

Simple add the Service Tag name as the addressPrefix. This will when deploying the UDR set it to use Service Tag and set it to the correct one.

APIM_UDR

The bicep resources would look something like this:

resource routeTable 'Microsoft.Network/routeTables@2023-04-01' = {
name: 'rt-${appPrefix}-${appName}'
  location: location
  properties: {
    disableBgpRoutePropagation: false
    routes: []
  }
}

resource route 'Microsoft.Network/routeTables/routes@2023-04-01' = {
  name: 'ApiManagement'
  parent: routeTable
  properties: {
    addressPrefix: 'ApiManagement'
    nextHopType: 'Internet'
    hasBgpOverride: false
  }
}
 
Happy deployment!
 

References

Private Endpoint Overview (GitHub) (Go here in case there are updates…)
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/private-link/private-endpoint-overview.md


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

The complete list of groupIds for private endpoint & privatelink service connection


azure

When creating a private link using ARM or Bicep, you need to specify a few settings, one of which are groupId for the privateLinkServiceConnections of the resource type you are connecting it to.  You can’t omit this value and it has to be exactly correct in order for the deployment to succeed, it is even case sensitive.

Example:

resource myPrivateEndpoint 'Microsoft.Network/privateEndpoints@2022-09-01' = {
name: '${webAppName}-privateendpoint'
location: location
properties: {
  subnet: {
    id: resourceId(networking.vNetResourceGroup, 'Microsoft.Network/virtualNetworks/subnets', networking.existingVNetName, privateLinkSubnet)
  }
  privateLinkServiceConnections: [
    {
      name: '${appService}-privateLink'
      properties: {
        privateLinkServiceId: appService.id
        groupIds: [
          'sites'
        ]
      }
    }
  ]
}
dependsOn: [
  appService
]
}

I have scoured the earth to find a list of them, the documentation of private endpoint bicep helpfully states that it is a ‘string’, but leaves out the values for the different resource types that you can connect to. This is all the documentation tells you:

PrivateEndpointIPConfigurationProperties

Name Description Value
groupId The ID of a group obtained from the remote resource that this private endpoint should connect to. string
PrivateLinkServiceConnectionProperties
Name Description Value
groupIds The ID(s) of the group(s) obtained from the remote resource that this private endpoint should connect to. string[]

https://learn.microsoft.com/en-us/azure/templates/microsoft.network/privateendpoints?pivots=deployment-language-bicep

There’s also a number of examples that helps find the groupid for sql, blob, table sites and a few more. But every time you need it for a new bicep, you have to look and look for the correct value.

Today, I spent some time not giving up, I started collecting every groupid I could find in a list, and after finding maybe 15 different ones, I got real lucky. Via a series of related searches I got to the key word ‘ListSupportedPrivateLinkResources’. This took me to a public GitHub repo named ‘Private Endpoint Overview’. A bit down on this page, there is a table called ‘private link resource’. This lists in turn: Private-link resource name, Resource type & Subresources. Subresources is the goldmine!

This is the (complete?) list of resource types that can have a Private Endpoint and their groupid’s

Private-link resource name Resource type Subresources
Azure API Management Service Microsoft.ApiManagement/service Gateway
Azure App Configuration Microsoft.Appconfiguration/configurationStores configurationStores
Azure Automation Microsoft.Automation/automationAccounts Webhook, DSCAndHybridWorker
Azure Cosmos DB Microsoft.AzureCosmosDB/databaseAccounts SQL, MongoDB, Cassandra, Gremlin, Table
Azure Batch Microsoft.Batch/batchAccounts batchAccount, nodeManagement
Azure Cache for Redis Microsoft.Cache/Redis redisCache
Azure Cache for Redis Enterprise Microsoft.Cache/redisEnterprise redisEnterprise
Azure Cognitive Services Microsoft.CognitiveServices/accounts account
Azure Managed Disks Microsoft.Compute/diskAccesses managed disk
Azure Container Registry Microsoft.ContainerRegistry/registries registry
Azure Kubernetes Service – Kubernetes API Microsoft.ContainerService/managedClusters management
Azure Data Factory Microsoft.DataFactory/factories dataFactory
Azure Data Explorer Microsoft.Kusto/clusters cluster
Azure Database for MariaDB Microsoft.DBforMariaDB/servers mariadbServer
Azure Database for MySQL Microsoft.DBforMySQL/servers mysqlServer
Azure Database for PostgreSQL – Single server Microsoft.DBforPostgreSQL/servers postgresqlServer
Azure Device Provisioning Service Microsoft.Devices/provisioningServices iotDps
Azure IoT Hub Microsoft.Devices/IotHubs iotHub
Azure IoT Central Microsoft.IoTCentral/IoTApps IoTApps
Azure Digital Twins Microsoft.DigitalTwins/digitalTwinsInstances API
Azure Event Grid Microsoft.EventGrid/domains domain
Azure Event Grid Microsoft.EventGrid/topics topic
Azure Event Hub Microsoft.EventHub/namespaces namespace
Azure HDInsight Microsoft.HDInsight/clusters cluster
Azure API for FHIR (Fast Healthcare Interoperability Resources) Microsoft.HealthcareApis/services fhir
Azure Key Vault HSM (hardware security module) Microsoft.Keyvault/managedHSMs HSM
Azure Key Vault Microsoft.KeyVault/vaults vault
Azure Machine Learning Microsoft.MachineLearningServices/workspaces amlworkspace
Azure Migrate Microsoft.Migrate/assessmentProjects project
Application Gateway Microsoft.Network/applicationgateways application gateway
Private Link service (your own service) Microsoft.Network/privateLinkServices empty
Power BI Microsoft.PowerBI/privateLinkServicesForPowerBI Power BI
Microsoft Purview Microsoft.Purview/accounts account
Microsoft Purview Microsoft.Purview/accounts portal
Azure Backup Microsoft.RecoveryServices/vaults AzureBackup, AzureSiteRecovery
Azure Relay Microsoft.Relay/namespaces namespace
Azure Cognitive Search Microsoft.Search/searchServices searchService
Azure Service Bus Microsoft.ServiceBus/namespaces namespace
Azure SignalR Service Microsoft.SignalRService/SignalR signalr
Azure SignalR Service Microsoft.SignalRService/webPubSub webpubsub
Azure SQL Database Microsoft.Sql/servers SQL Server (sqlServer)
Azure Storage Microsoft.Storage/storageAccounts Blob (blob, blob_secondary)
Table (table, table_secondary)
Queue (queue, queue_secondary)
File (file, file_secondary)
Web (web, web_secondary)
Dfs (dfs, dfs_secondary)
Azure File Sync Microsoft.StorageSync/storageSyncServices File Sync Service
Azure Synapse Microsoft.Synapse/privateLinkHubs web
Azure Synapse Analytics Microsoft.Synapse/workspaces Sql, SqlOnDemand, Dev
Azure App Service Microsoft.Web/hostingEnvironments hosting environment
Azure App Service Microsoft.Web/sites sites
Azure App Service Staging slot Microsoft.Web/sites sites-staging (undocumented?)
Azure Static Web Apps Microsoft.Web/staticSites staticSites
Azure Media Services Microsoft.Media/mediaservices keydelivery, liveevent, streamingendpoint
Resource Management Private Links Microsoft.Authorization/resourceManagementPrivateLinks ResourceManagement
Azure Databricks Microsoft.Databricks/workspaces databricks_ui_api, browser_authentication
Azure Monitor Private Link Scope Microsoft.Insights/privatelinkscopes azuremonitor
 
Happy deployment!
 

References

Private Endpoint Overview (GitHub) (Go here in case there are updates…)
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/private-link/private-endpoint-overview.md


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

Export all data from Azure Cosmos DB for MongoDB to file for archiving


azure

If you want to export all data from a MongoDB in Azure, then you have come to the right place.

Maybe there is a better way, but I could not find it. Azure data studio allows me to connect, and in a query to the collection get data, but it only gets me about 30 items at the time, so copy paste from the results in not realisitc.

This procedure, is done on a WIndows environment, with internet connectivity, the MongoDB does not have any IP restrictions, if it does, you would have to fix connectivity first. I guess a good test can be to connect using the Azure Data Studio.

Here it goes!

mongo2

mongoexport /h mymongodbaccount.mongo.cosmos.azure.com:10255 /u mymongodbaccount /p "RANDOMCHARACTERSRANDOMCHARACTERSRANDOMCHARACTERSRANDOMCHARACTERSRANDOMCHARACTERS==" /ssl /d myDatabase /c myDataCollection /o c:\Temp\myDataCollection.json --authenticationDatabase "admin"

Broken down, this command has the following components:

mongoexport

command line .exe tool that is part of the mongodb database tools package that is a free download from mongodb.com, see link in the reference section

/h mymongodbaccount.mongo.cosmos.azure.com:10255

host, the complete path including the port to your MongoDB in Azure. You will find this string in the connection string section in the Azure portal

/u mymongodbaccount

User, this is the name of the Cosmos DB for MongoDB account, also the first section of the host string.

/p “RANDOMCHARACTERSRANDOMCHARACTERSRANDOMCHARACTERSRANDOMCHARACTERS==”

Password, this is also find in the connection string section (I could be wrong here, I deleted all my DB’s before creating this post…if not here, then its in a different section in the portal, it IS there though)

/ssl

LTS/SSL/Https, secure encrypted connection

/d myDatabase

Database, the database name where you are exporting from

/c myDataCollection

Collection, the name of the collection you are exporting from. The file will contain all data from this collection in json format

/o c:\Temp\myDataCollection.json

Output/target file, put it anywhere locally on your computer

–authenticationDatabase “admin”

This is required for the auth using the account + password to work

 

The result, when executed in a CMD prompt looks like this:

mongo

 

Important!
If you have more than one collection, in more than one mongoDB, then you will have to run the command again and again, remember though to change the output filename.

 
Happy exporting!
 

References

https://www.mongodb.com/try/download/database-tools

https://www.mongodb.com/community/forums/t/unable-to-authenticate-using-mechanism-scram-sha-1/86151/4

 


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

List all changes made in your Azure environment in one query using Azure Resource Graph


azure

The kusto query below is using the new ResourceChanges resource, and will give you a list of all changes made in your Azure environment, create, update, delete, automatic or manual, it will all be there. (Where you have access).
This is a great way to keep track of what is happening in and to your environment.

You will get the following info from each change made:

Subcription Name
ChangeTime
ResourceName
ResourceType
ResourceGroup
ChangeType
SubscriptionName
SubscriptionId
TargetResourceId
CorrelationId
ChangeCount
ChangedProperties

This list can get pretty long, so it can be a good idea to filer and sort a bit. Especially on the timeframe (in the sample query, it is set to 7 days).
Filter can be applies to subscription, dates, resource groups, resource name, type, etc.
Example:

| where resourceGroup contains “production”
Will give you only the changes made to RG’s with production in the name.

| where resourceType = “virtualMachines”
Will give you only the changes made to resources of type Virtual Machine.

Open the Azure portal, find ‘Azure Resource Graph Explorer’, then paste the following query in the query windows and hit run.

ResourceChanges
| join kind=inner
   (resourcecontainers
   | where type == 'microsoft.resources/subscriptions'
   | project subscriptionId, subscriptionName = name)
   on subscriptionId
| extend changeTime = todatetime(properties.changeAttributes.timestamp), targetResourceId = tostring(properties.targetResourceId),
changeType = tostring(properties.changeType), correlationId = properties.changeAttributes.correlationId,
changedProperties = properties.changes, changeCount = properties.changeAttributes.changesCount
| extend resourceName = tostring(split(targetResourceId, '/')[-1])
| extend resourceType = tostring(split(targetResourceId, '/')[-2])
| where changeTime > ago(7d)
// Change the time span as preferred, 1d(1 day/24h), 7d, 30d...
| where subscriptionName contains "DevTest" // "" for all subscriptions
| order by changeType asc, changeTime desc
// Change what you sort by as prefered, type, time, subscriptionName, etc.
| project changeTime, resourceName, resourceType, resourceGroup, changeType, subscriptionName, subscriptionId, targetResourceId, 
correlationId, changeCount, changedProperties
ResourceChanges1
As this is a functionality in public preview at the time of posting, the following applies:

Important!
Resource configuration changes is in Public Preview and only supports changes to resource types from the Resources table in Resource Graph. This does not yet include changes to the resource container resources, such as Management groups, Subscriptions, and Resource groups.

Happy resource mining!

References

https://docs.microsoft.com/en-us/azure/governance/resource-graph/how-to/get-resource-changes


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

List all Linux VM extensions in one query using Azure Resource Graph


azure

The kusto query below will give you a list of all Linux VMs and the extensions they have…
It was handy when the OMI vulnerability was out.

You will get the following info:

VM Name
Operating System
Extensions
Add additional columns to get more info.

Resources
| where type == 'microsoft.compute/virtualmachines'
| extend
    JoinID = toupper(id),
    OSName = tostring(properties.osProfile.computerName),
    OSType = tostring(properties.storageProfile.osDisk.osType),
    VMSize = tostring(properties.hardwareProfile.vmSize)
| join kind=leftouter(
    Resources
    | where type == 'microsoft.compute/virtualmachines/extensions'
    | extend
        VMId = toupper(substring(id, 0, indexof(id, '/extensions'))),
        ExtensionName = name
) on $left.JoinID == $right.VMId
| where OSType == 'Linux'
| summarize Extensions = make_list(ExtensionName) by OSName, OSType
| order by tolower(OSName) asc
The result looks something like this:
LinuxKusto
Happy resource mining!

References
https://docs.microsoft.com/en-us/azure/governance/resource-graph/samples/advanced?tabs=azure-cli


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

List all UDR User Defined Routes in one query using Azure Resource Graph


azure

The kusto query below will give you a list of all manually added routes/UDRs in all of your Route Tables(RT) in all subnets in all subscriptions. (Where you have access).
This is a great way to keep track of your UDRs…lest they get out of hand…

You will get the following info from each UDR:

Subcription Name
Resource Group Name
Subnet Name
RT Name
UDR Name
Target Prefix
Next Hop Type
Next Hop IP Address
Provisioning State
Has BGP Override

In my current Azure network, the total UDR count is around 413, in 39 different RTs. Its not easy to keep track and t-shoot the network, not easy at all.

Use different sort or where clauses to filter and sort on what you are currently looking for, if you for example filter on
| where addressPrefix == ‘0.0.0.0/0’
you will see only the routes for ‘unrouted’ traffic.
| where nextHopType == ‘VirtualAppliance’
will list all routes to your(?) Virtual appliance Firewall, and so on.
(You can add the where clauses at the end below the last line)

resources
| where type =~ "Microsoft.Network/routeTables"
| mv-expand rules = properties.routes
| join kind=leftouter (resourcecontainers 
| where type=='microsoft.resources/subscriptions' 
| project SubcriptionName=name, subscriptionId) on subscriptionId
| extend subnet_name = split((split(tostring(properties.subnets), '/'))[10], '"')[0]
| extend addressPrefix = tostring(rules.properties.addressPrefix)
| extend nextHopType = tostring(rules.properties.nextHopType)
| extend nextHopIpAddress = tostring(rules.properties.nextHopIpAddress)
| extend hasBgpOverride = tostring(rules.properties.hasBgpOverride)
| extend provisioningState = tostring(rules.properties.provisioningState)
| extend udrname = rules.name
| extend rtname = name
| project SubcriptionName, resourceGroup, subnet_name, rtname, udrname, addressPrefix, nextHopType, nextHopIpAddress, provisioningState, hasBgpOverride
| sort by SubcriptionName, resourceGroup asc, rtname asc, addressPrefix asc
The result looks something like this:
UDRKusto
Happy resource mining!

References
https://docs.microsoft.com/en-us/azure/governance/resource-graph/samples/advanced?tabs=azure-cli


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

Error: Policies attempted to append some fields which already exist in the request with different values


azure

In our current environment, this is starting to show up more and more.
We have a number of policies in Azure, among them are ‘resources-tag-enforcement’, the one that adds a number of required Tags on resources on our production Resource Groups. Because we have them and they are required on all resources on the groups, things fail.
It is changes that start to fail, unrelated to the Tag policy really…
Example 1: We tried to change pricing tier on an App Service plan – Fail
Example 2: We deleted a rule in Azure Firewall – Fail
Example 3: We resized a VM – Fail

All due to one thing, the Resource Manager seems to think in the backend, that these changes are really not just changes, but we are adding new resources to the resource groups, triggering the policy, and that fails to append the Tags since the Tags are already there…
Same scenario with a ‘Allowed location’ policy, if a resource was once created outside of the allowed scope, changes are not allowed since the RM seems to think that you are really adding new resources, not just making adjustments.

The error you will see from the Tag policy is this:

Failed to resize virtual machine ‘XXX-NNNN’ to size ‘Standard DXXX’. Error: Policies attempted to append some fields which already exist in the request with different values. Fields: ‘tags[nnn1],tags[nnn2]’. Policy identifiers:'[{“policyAssignment”:{“name”:”Assignment – resources-tag-enforcement”,”id”:”/subscriptions/nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn/providers/Microsoft.Authorization/policyAssignments/Assignment – resources-tag-enforcement”},”policyDefinition”:{“name”:”resources-tag-enforcement”,”id”:”/subscriptions/nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn/providers/Microsoft.Authorization/policyDefinitions/resources-tag-enforcement”}}]’. Please contact the subscription administrator to update the policies.
 
Resolution:
Temporarily Disable the assignment of the policy. Look up the Resource Group, go to policys, assignments, then the correct policy and open it.
A bit down, you have to switch to disable it.
WindowsUpdate2
Set it to Disabled and Review+Save, then Save.
Done.
Remember to enable it again after you are done.
 
 
Good luck!
 
References


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

Blocked traffic on a public IP on a Load balancer


azure

Issue:

You setup a new Load balancer with public IP(s) for incoming traffic, but when testing, all incoming traffic is blocked. No reason, its just blocked.

(Resolution at the end…)

Scenario:
Azure based environment.
An appliance (or a VM) with one or two NIC’s connected to a public IP

Resolution:
Add a NSG to the public IP’s…and specifically allow the traffic you want to allow, if you don’t care, add from any to any port *
If you don’t add an NSG to the NIC, traffic to the Public IP will be blocked…it just is.

Azure Firewall SNAT private IP address ranges
https://technet.microsoft.com/en-us/library/mt683473(v=office.16).aspx

Deploy an Azure Firewall with multiple public IP addresses using Azure PowerShell
https://docs.microsoft.com/en-us/azure/firewall/deploy-multi-public-ip-powershell

Thanks to:
Thomas Vuylsteke – Microsoft Azure Fasttrack team
Microsoft Premier Support
Akelius Residential Property AB (Martin Supan, Mattias Segerström)


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn

Error: Policies attempted to append some fields which already exist in the request with different values


azure

In our current environment, this is starting to show up more and more.
We have a number of policies in Azure, among them are ‘resources-tag-enforcement’, the one that adds a number of required Tags on resources on our production Resource Groups. Because we have them and they are required on all resources on the groups, things fail.
It is changes that start to fail, unrelated to the Tag policy really…
Example 1: We tried to change pricing tier on an App Service plan – Fail
Example 2: We deleted a rule in Azure Firewall – Fail
Example 3: We resized a VM – Fail

All due to one thing, the Resource Manager seems to think in the backend, that these changes are really not just changes, but we are adding new resources to the resource groups, triggering the policy, and that fails to append the Tags since the Tags are already there…
Same scenario with a ‘Allowed location’ policy, if a resource was once created outside of the allowed scope, changes are not allowed since the RM seems to think that you are really adding new resources, not just making adjustments.

The error you will see from the Tag policy is this:

Failed to resize virtual machine ‘XXX-NNNN’ to size ‘Standard DXXX’. Error: Policies attempted to append some fields which already exist in the request with different values. Fields: ‘tags[nnn1],tags[nnn2]’. Policy identifiers:'[{“policyAssignment”:{“name”:”Assignment – resources-tag-enforcement”,”id”:”/subscriptions/nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn/providers/Microsoft.Authorization/policyAssignments/Assignment – resources-tag-enforcement”},”policyDefinition”:{“name”:”resources-tag-enforcement”,”id”:”/subscriptions/nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn/providers/Microsoft.Authorization/policyDefinitions/resources-tag-enforcement”}}]’. Please contact the subscription administrator to update the policies.
 
Resolution:
Temporarily Disable the assignment of the policy. Look up the Resource Group, go to policys, assignments, then the correct policy and open it.
A bit down, you have to switch to disable it.
WindowsUpdate2
Set it to Disabled and Review+Save, then Save.
Done.
Remember to enable it again after you are done.
 
 
Good luck!
 
References


___________________________________________________________________________________________________

Enjoy!

Regards

 Thomas Odell Balkeståhl on LinkedIn