Use Azure Key Vault Store of Wild card certificate created via certbot and update the local nginx webserver and Azure Application Gateway

A follow on from the Azure wildcard certbot certificate with Azure DNS TXT updates, this part is store the wild card SSL certificate that certbot created and place into azure key vault.

I am going to break this down into 2 parts So, to start with I create a script to basically use on your linux server and the second is via the application gateway

  1. clean out any old processing of the certificates
  2. pull down the certificate from the azure key vault store
  3. convert the PFX certificate into the
    1. full chain
    2. private key
  4. update the nginx hosting certificate files
  5. reload nginx

So here are the steps in full, I am using a sub directory called “certs” to process the certificate

echo “clean out the certs”
rm certs/*

Change the following to your azure key vault and wildcard certificate name, please note we have to use the encoding base64

az keyvault secret download –file certs/wild.pfx –id https://<azure key vault name>.vault.azure.net/secrets/<certificate name> –encoding base64

And now just convert that PFX file into the private key and then the full certificate file (I am channing the client certificates and the CA certificates into the full.crt file)

openssl pkcs12 -in certs/wild.pfx -passin pass: -nocerts -nodes -out certs/priv .key
openssl pkcs12 -in certs/wild.pfx -passin pass: -clcerts -nokeys -out certs/ful l.crt
openssl pkcs12 -in certs/wild.pfx -passin pass: -cacerts -nokeys -chain >> cer ts/full.crt

And then just push the certs folder into the nginx configured ssl directory that links to your nginx configure SSL options

cp certs/* /etc/nginx/ssl/<domain name>/

final part, reload nginx to use the new certificate!

systemctl reload nginx

To use on the Application Gateway — well that is very simple as just going to the application gateway -> listeners -> choose your listener you want to update and then choose the certificate from the key vault.

AND THAT IS IT 🙂 — if you need any more advice on certain areas please say!

Use Azure Application Gateway with the certbot wild card certificate stored within the Azure Key Vault

A follow on from the Azure wildcard certbot certificate with Azure DNS TXT updates, this part is to store the wild card SSL certificate that certbot created and place into azure key vault.

So in essence the process is to

  1. Check for new certificates via the certbot renewal process
  2. If there is a new certificate create a PFX
  3. Upload this PFX file into the key store

So, to start with lets obtain a new certificate (if there is one!!) , please change your domain name to the domain name that you using, the preferred challenge should be to whatever you have setup to be the default process for this certificate, I am using the azure DNS zone so using the azure certbot-azure-dns certbot plugin

certbot certonly –manual -d ‘*.<domain name>’ –preferred-challenges=dns

The next part is the most important part!, it is creating the PFX file from the new ticket! Please note you have to pass in the whole key chain e.g the chain / fullchain files. (change the domain name again to what you are using)

openssl pkcs12 -export -out <domain name>.pfx -inkey /etc/letsencrypt/live/<domain name>/privkey.pem -in /etc/letsencrypt/live/<domain name>/cert.pem  -in /etc/letsencrypt/live/<domain name>/cert.pem -in /etc/letsencrypt/live/<domain name>/chain.pem -in /etc/letsencrypt/live/<domain name>/fullchain.pem

And then the last part is to just upload that certificate into the azure key vault (you have to have enabled the managed identity for key vault of the service you are using)

az keyvault certificate import –vault-name “<key vault name>” -n “<certificate common name in the file store e.g my domin>” -f <domain name>.pfx

AND THAT IS IT 🙂 — if you need any more advice on certain areas please say!

Azure wildcard certbot certificate with Azure DNS TXT updates

The requirement was to set up a wildcard certificate on azure, so I used a nice tool called certbot that can generate single subdomain certificates or a wild card certificate.

For the creation of a wild car certificate, we need to be able to alter the DNS TXT records and via azure we can achieve this via the

  1. Install certbot onto a server (I am using a linux server)
  2. User role to only allow DNS TXT record updates
  3. Allow the managed identity (in essence the server that is altering the DNS TXT records)

So, to start things off lets install the certbot and I am using the snap package manager and also the nginx as the web server.

snap install core
snap refresh core
snap install –classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot

The certbot plugin to converse with the azure dns is certbot-dns-azure, as this isn’t part of the offical packages we have to use the “–edge” option below and also allow the certbot trust this plugin.

snap set certbot trust-plugin-with-root=ok
snap install –edge certbot-dns-azure
snap connect certbot:plugin certbot-dns-azure

For the certbot dns azure plugin to function automatically we need to create a file within the .azure folder called dns.ini ( e.g. /root/.azure/dns.ini or ~/.azure/dns.ini)

certbot renew –dry-run

To fill in the options below are as below (small hint! the subscription ID + resource group name could be obtained from the URL when you goto the dns zone within azure you want to use!)

  1. linux server ID = I got this after the creation of the role access
  2. dns cloud name = is the azure named resource of your dns zone
  3. subscriptionID = your subscription ID
  4. resource groupname = the groupname of resource

dns_azure_msi_client_id = <linux server ID>

dns_azure_zone1 = <dns cloud name>:/subscriptions/<subscriptionID>/resourceGroups/<resource groupname>/providers/Microsoft.Network/dnsZones/<dns cloudname>

The last part of this solution is to allow the managed identy (e.g. the linux server) to update the DNS TXT records for the azure dns zone.

Iif you goto your subscription -> access control (IAM) — On the left menu -> Add(top bar) custom role

In the new custom role, please define the basics e.g. name, but within the permissions use the “add permission” and include the one below (Microsoft.Network/dnszones/TXT

The json would be something akin to

{ "id": "/subscriptions/<subscriptionID>/providers/Microsoft.Authorization/roleDefinitions/<roleID>", 
"properties": { "roleName": "DNS TXT Contributor", "description": "User role only allows DNS TXT updates.", "assignableScopes": [ "/subscriptions/<subscriptionID>" ], "permissions": [ { "actions": [ "Microsoft.Network/dnszones/TXT/read", "Microsoft.Network/dnszones/TXT/write" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] }}

and then within the DNS Zone (your DNS configuration) go to Access Control (IAM) and then click “add role assignment” where the role will be the role that you created above

and then we just need to associate it (members) with your managed identities (e.g. your webserver!)

The next post (once I have created it, there will be a link here!) will be about how to create the certificate and then convert it into a PFX file for uploading into the azure key vault file storage.

AND THAT IS IT 🙂 — if you need any more advice on certain areas please say!

Azure devops git (ssh) config on linux

I am using fedora linux as my development environment OS, I love it but when you are working with azure that mainly believes you are using windows. Then you have to make some changes to your configuration files that windows (may do??) for you.

So, after I created my ssh key for the development, I keep on getting an issue to pull/push up my local git repo where the CLI would just hang or error out, so after doing the -v (verbose mode)

git pull -v

The issue highlighted itself with the following error message

Unable to negotiate with 51.104.26.0 port 22: no matching host key type found. Their offer: ssh-rsa

So, all I did was to update the ~/.ssh/config (your local username home directory .ssh config(uration) file. Please note the last bits, the HostkeyAlgorithms and PubkeyAcceptedKeyTypes

Host ssh.dev.azure.com
       PreferredAuthentications publickey
       IdentityFile ~/your/key
       UpdateHostKeys no
       IdentitiesOnly yes
       HostkeyAlgorithms +ssh-rsa
       PubkeyAcceptedKeyTypes +ssh-rsa,rsa-sha2-256,rsa-sha2-512

After that, there were no issues, I suppose it is the classic of reading the error message!

Azure gateway issues with wild card certificates

Within my current company, we are using azure application gateway to host the application and it is a nice load balancer but with a let’s encrypt wild card certificate I kept on getting the following message on the “Backend health” page where we were trying to use https from the application gateway to the server (keeping things secure is always nice 🙂 )

The Common Name (CN) of the backend server certificate does not match the host header entered in the health probe configuration (v2 SKU) or the FQDN in the backend pool (v1 SKU). Verify if the hostname matches with the CN of the backend server certificate. To learn more visit - https://aka.ms/backendcertcnmismatch.

The main problem is using a wild card certificate e.g. for example.com and a listener pointing to a single subdomain e.g. test.example.com, is that we have to set up “Health probes” to confirm that the backend server was actually hosting test.example.com instead of checking example.com.

So, the nginx daemon was running on the linux server using the wild card certificate we generated via azure functions to generate the let’s encrypt certificate using the hostname of test.example.com and then setup a health probe on the application gateway as below.

and then the health probe will start to work instead of giving the error above, below are the health probes with a status of 200

If the above doesn’t help, just shout.