Deploying to Azure Cloud using Spinnaker

Purpose

Spinnaker as a renowned Continuous Delivery tool supports app deployment into all major cloud environments including Azure cloud. Here, We guide you through the configuration and pipeline setup of an app into Azure Cloud using Spinnaker.

Assumptions

We should be having the following items working/configured before configuring Spinnaker…
  • Valid Azure Subscription/Account Spinnaker 1.16.2
  • Working Jenkins integrated with Spinnaker. Jenkins does your CI build
  • Our app is a Java springboot application, to be deployed into Azure cloud. These inputs are quired for configuring Load Balancer and Firewall
    – App is listening on port 8080
    – Healthcheck url is http://localhost:8080/health
    – App url is http://localhost:8080/greeting

Preparing Azure for Spinnaker Integration

  • Azure CLI for creating Azure resources for Spinnaker
  • Azure Portal (portal.azure.com) for Virtual network setup and Action monitoring

Installing Azure CLI

Azure CLI is required to set up pre-requirements on the Azure cloud.

Install Azure CLI on your Laptop or any VM which has connectivity to the Azure cloud. Simply run this command

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

For detailed manual instruction, you can follow the guide from the link
https://docs.microsoft.com/en-gb/cli/azure/install-azure-cli-apt?view=azure-cli-latest

Create Azure resources for Spinnaker Integration

In your Azure account, essentially you will need a few items created…

  • Service Principle (Halyard to communicate with Azure),
  • Resource Group (All of the Azure resources like temp VMs, machine image, etc to be grouped under this name)
  • Azure Vault (Default Username and password/key of Azure VMs created Spinnaker are stored here)
    – SSH Username (Default username of VM)
    – SSH Password (Default password of VM

Throughout the procedure here, I choose to configure and deploy Azure resources in eastus region. But you can change it as per your choice.

Configure Azure CLI to create Azure resources

1. Login to Azure by Azure CLI

saga@sagamc:~$ az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
Opening in existing browser session.
You have logged in. Now let us find all the subscriptions to which you have access...
[
  {
    "cloudName": "AzureCloud",
    "id": "959c83bf-0f16-4a56-a542-2067be19692e", #Subscription ID 
     "isDefault": true,
    "name": "Microsoft Azure Sponsorship",
    "state": "Enabled",
    "tenantId": "d1ea7473-64d0-44ed-a093-ad8041071ee6",
    "user": {
      "name": "sagayaraj.d@opsmx.io",
      "type": "user"
    }
  }
]

2. Get your Subscription details

Note: You can also get your subscription from Azure portal: Home -> Subscriptions

saga@sagamc:~$ az account list
[
  {
    "cloudName": "AzureCloud",
    "id": "959c83bf-0f16-4a56-a542-2067be19692e", #Subscription ID
    "isDefault": true,
    "name": "Microsoft Azure Sponsorship", #Subscription Name
    "state": "Enabled",
    "tenantId": "d1ea7473-64d0-44ed-a093-ad8041071ee6",
    "user": {
      "name": "sagayaraj.d@opsmx.io",
      "type": "user"
    }
  }
]

3. Make your Azure CLI to work with your Subscription.

saga@sagamc:~$ SUBSCRIPTION_ID=959c83bf-0f16-4a56-a542-2067be19692e
saga@sagamc:~$ az account set --subscription $SUBSCRIPTION_ID

Create Azure resources
4. Create a Service Principal. This is the account by which Spinnaker’s Halyard will communicate with Azure cloud.

Syntax: az ad sp create-for-rbac --name "<ServicePrincipalName>"

saga@sagamc:~$ az ad sp create-for-rbac --name "Spinnaker-SP"
Changing "Spinnaker-SP" to a valid URI of "http://Spinnaker-SP", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/959c83bf-0f16-4a56-a542-2067be19692e"
  Retrying role assignment creation: 1/36
  Retrying role assignment creation: 2/36
  Retrying role assignment creation: 3/36
{
  "appId": "fc061b7e-d5c5-474d-a888-f34fd69bd85b",
  "displayName": "Spinnaker-SP",
  "name": "http://Spinnaker-SP",
  "password": "1fd416e8-15e2-4795-8a6e-aea4eabca3be",
  "tenant": "d1ea7473-64d0-44ed-a093-ad8041071ee6"
}

5. Make a note of AppId and TenantId

saga@sagamc:~$ APP_ID=fc061b7e-d5c5-474d-a888-f34fd69bd85b
saga@sagamc:~$ TENANT_ID=d1ea7473-64d0-44ed-a093-ad8041071ee6

6. Create a resource group in eastus region.

az account list-locations --query [].name
Syntax: az group create --name <resourcegroup-name> --location <location>
saga@sagamc:~$ RESOURCE_GROUP=SpinHal-RG
saga@sagamc:~$ az group create --name $RESOURCE_GROUP --location eastus
{
  "id": "/subscriptions/959c83bf-0f16-4a56-a542-2067be19692e/resourceGroups/SpinHal-RG",
  "location": "eastus",
  "managedBy": null,
  "name": "SpinHal-RG",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

7. Create a Azure vault. The vault will store default username and password of VMs.

Note: Vault names are used in a DNS record (like spinnakervault.vault.azure.net), so the name must be unique across all customers.

saga@sagamc:~$ VAULT_NAME=SpinHal-Vault
saga@sagamc:~$ az keyvault create --enabled-for-template-deployment true --resource-group $RESOURCE_GROUP --name $VAULT_NAME
{
  "id": "/subscriptions/959c83bf-0f16-4a56-a542-2067be19692e/resourceGroups/SpinHal-RG/providers/Microsoft.KeyVault/vaults/SpinHal-Vault",
  "location": "eastus",
  "name": "SpinHal-Vault",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "9aeada99-4d00-4f53-8567-956c8645e905",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "d1ea7473-64d0-44ed-a093-ad8041071ee6"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": true,
    "networkAcls": null,
    "provisioningState": "Succeeded",
    "sku": {
      "name": "standard"
    },
    "tenantId": "d1ea7473-64d0-44ed-a093-ad8041071ee6",
    "vaultUri": "https://spinhal-vault.vault.azure.net/"
  },
  "resourceGroup": "SpinHal-RG",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

8. Set Vault permissions

saga@sagamc:~$ az keyvault set-policy --secret-permissions get --name $VAULT_NAME --spn $APP_ID
{
  "id": "/subscriptions/959c83bf-0f16-4a56-a542-2067be19692e/resourceGroups/SpinHal-RG/providers/Microsoft.KeyVault/vaults/SpinHal-Vault",
  "location": "eastus",
  "name": "SpinHal-Vault",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "9aeada99-4d00-4f53-8567-956c8645e905",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "d1ea7473-64d0-44ed-a093-ad8041071ee6"
      },
      {
        "applicationId": null,
        "objectId": "05676264-65c8-48c7-925d-00daba4bc5d6",
        "permissions": {
          "certificates": null,
          "keys": null,
          "secrets": [
            "get"
          ],
          "storage": null
        },
        "tenantId": "d1ea7473-64d0-44ed-a093-ad8041071ee6"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": true,
    "networkAcls": null,
    "provisioningState": "Succeeded",
    "sku": {
      "name": "standard"
    },
    "tenantId": "d1ea7473-64d0-44ed-a093-ad8041071ee6",
    "vaultUri": "https://spinhal-vault.vault.azure.net/"
  },
  "resourceGroup": "SpinHal-RG",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

9. Insert the default SSH username of VMs into your Azure vault

saga@sagamc:~$ VMUSER=ddsaga
saga@sagamc:~$ az keyvault secret set --name VMUsername --value $VMUSER --vault-name $VAULT_NAME 
{
  "attributes": {
    "created": "2019-11-14T03:12:18+00:00",
    "enabled": true,
    "expires": null,
    "notBefore": null,
    "recoveryLevel": "Purgeable",
    "updated": "2019-11-14T03:12:18+00:00"
  },
  "contentType": null,
  "id": "https://spinhal-vault.vault.azure.net/secrets/VMUsername/a7da593552874f76afef83cfd11d8232",
  "kid": null,
  "managed": null,
  "tags": {
    "file-encoding": "utf-8"
  },
  "value": "ddsaga"
}

10. Set public key in Vault

a. Generate SSH pubkey to insert into Vault

saga@sagamc:~$ ssh-keygen -f ddsaga -N ''
#Username is ddsaga (Change it as per your choice)
#cat ddsaga.pub and copy the content without the last part <user>@<host> [e.g. opsmxuser@spinnaker-saga]

saga@sagamc:~$ VMUSER_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMGBDPxjYydvel/ZyYoaLM3zDB9tGY+YbkcpbnI/esqR2qGgs/rG9C1MWP3ews3d2F58Ss2F7C03tj+5phbhB+10tmnPn2ezAq3FXcjEenQluVb2QInxnr+bdfDeZLyYcApFemgs0mgU5LcP2ixGQ6Tli9xyziITRuaIr904pIOcbdo8+OLDXFS7W0+KhSu3RjCkD4EWNpbCB48j+Oz8YD6t6X5aixEjFg7v4Swo0+NxAeFteCyzU8hWZAdVW0eWdjo5zeQosMW7tVe+UFBDVgQfjrZd0ue0xOyYRf1HLplIEcyJpjy2DhWbG99FJW9gdbJqxeGLop7lKvbatrpBMp"

b. Insert the default SSH pubkey of VMs into your Azure vault

saga@sagamc:~$ az keyvault secret set --name VMSshPublicKey --vault-name $VAULT_NAME --value "$VMUSER_KEY"
{
  "attributes": {
    "created": "2019-11-14T03:21:02+00:00",
    "enabled": true,
    "expires": null,
    "notBefore": null,
    "recoveryLevel": "Purgeable",
    "updated": "2019-11-14T03:21:02+00:00"
  },
  "contentType": null,
  "id": "https://spinhal-vault.vault.azure.net/secrets/VMSshPublicKey/38ab80bc184d4beab03e35dff30c9c8f",
  "kid": null,
  "managed": null,
  "tags": {
    "file-encoding": "utf-8"
  },
  "value": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMGBDPxjYydvel/ZyYoaLM3zDB9tGY+YbkcpbnI/esqR2qGgs/rG9C1MWP3ews3d2F58Ss2F7C03tj+5phbhB+10tmnPn2ezAq3FXcjEenQluVb2QInxnr+bdfDeZLyYcApFemgs0mgU5LcP2ixGQ6Tli9xyziITRuaIr904pIOcbdo8+OLDXFS7W0+KhSu3RjCkD4EWNpbCB48j+Oz8YD6t6X5aixEjFg7v4Swo0+NxAeFteCyzU8hWZAdVW0eWdjo5zeQosMW7tVe+UFBDVgQfjrZd0ue0xOyYRf1HLplIEcyJpjy2DhWbG99FJW9gdbJqxeGLop7lKvbatrpBMp"
}

That brings up the end of activities on Azure CLI.

Till this point, all the above steps can be executed on any machine

Integrating Azure provider in Spinnaker

Use Halyard command-line program to enable the integration of Azure in Spinnaker.

The Procedure below should be done on hal machine (VM/Pod/Container)

1. During the above procedure “Create Azure resources for Spinnaker Integration“, you would have created resources like Service Principle, Resource Group and Azure Vault. Make aNote those values and set them as env variables here

spinnaker@hal-xyz:~$ cat > azureapp.vars <<-EOF
SUBSCRIPTION_ID=959c83bf-0f16-4a56-a542-2067be19692e
APP_ID=fc061b7e-d5c5-474d-a888-f34fd69bd85b
TENANT_ID=d1ea7473-64d0-44ed-a093-ad8041071ee6
RESOURCE_GROUP="SpinHal-RG"
VAULT_NAME=SpinHal-Vault 
EOF

spinnaker@hal-xyz:~$ source azureapp.vars

Optionally, you can verify the variables are set correctly

echo $APP_ID
echo $TENANT_ID
echo $SUBSCRIPTION_ID
echo $VAULT_NAME
echo $RESOURCE_GROUP

2. Execute the ‘hal’ command to configure hal with Azure. This step will prompt you for password. The password was outputted for you when Service principle was created via Azure CLI.

spinnaker@hal-xyz:~$ hal config provider azure account add saga-azure-account \
--client-id $APP_ID \
--tenant-id $TENANT_ID \
--subscription-id $SUBSCRIPTION_ID \
--default-key-vault $VAULT_NAME \
--default-resource-group $RESOURCE_GROUP \
--packer-resource-group $RESOURCE_GROUP \
--app-key

3. Enable Azure provider in Spinnaker

spinnaker@hal-xyz:~$ hal config provider azure enable

4. You may review the Azure provider configuration

spinnaker@hal-xyz:~$ hal config provider azure account list
saga-azure-account #Output would contain this value. Use this value in the next command

spinnaker@hal-xyz:~$ hal config provider azure account get saga-azure-account

#Enable eastus region. The region should match the region configured in the earlier
Azure CLI steps

spinnaker@hal-xyz:~$ hal config provider azure account edit saga-azure-account - -
regions=eastus

5. Complete the hal configuration with “hal deploy apply” command.

Note: Now if you create a new application in Spinnaker, the application should have the provider
‘azure’ in the list.

Azure Virtual Network

Spinnaker will deploy applications on a Virtual network in Azure cloud. Hence, ensure to have a Virtual network and minimum one Subnet created in eastus region (or other region, it should match hal configuration). You may create the virtual network on portal.azure.com

My Virtual network details are…

Network name: Spin_Vnet
CIDR: 172.16.0.0/22
Subnet: default

The Virtual network and Subnet will be used by Firewall, Load Balancer and Server group configurations.

Azure and Spinnaker Resource Mapping

Spinnaker generalizes the Cluster terms across multiple Cloud. The resource in Spinnaker could mean different thing in Azure cloud.

  • Spinnaker Firewall means Network Security Goup in Azure cloud.
  • Spinnaker Load Balancer is referred as Load Balancer only in  Azure.
  • Spinnaker Server Group is referred as Virtual Machine ScaleSet in Azure.

Configuring Azure application in Spinnaker UI

Note: Perform these steps in Spinnaker UI.

1. Create an Application

Under Applications, Create an application ‘appxinazure’. Ensure to select ‘azure‘ in the provider list.

2. Configure Application’s Infrastructure

Prior to configuring a CD pipeline, we need to configure the Infrastructure of the application. A separate resource group by name <appname>-<region> will be created in Azure cloud automatically. In our case ‘appxinazure-eastus’ is the new resource group. The resources created in the Spinnaker UI like Firewall, Load Balancer, VM ScaleSet are stored on this resource group; however the baked images are stored on the resource group configured in Halyard.

** To successfully configure the Firewall, Load Balancer and Server group – Virtual network and Subnet are one of the pre-requirement. You should have them created. If not, refer the section Azure Virtual Network to setup one.

a. Create Firewall

Create a firewall first. This firewall settings are inheritted by the VMs. So whiteliest the port 8080 (App is listening on it) for LoadBalancer to reach this port; Similarly whitelist the port 22 for any SSH session that may be required for debugging.

b. Create Load balancer

You should know the application listening port, health check page, app url, so that the load balancer can be configured correctly.


c. Create Server Group

Do not create server group using the Infrastructure tab, because you may be unsuccessfull – while it will look for a image that will not available until we perform a Bake stage. Hence, configuring server group is part of the pipeline setup process. You will have to create server group in Deploy stage.

Configuring Pipeline

Under Pipeline section, click ‘Create’ to create a New Pipeline with type being selected as Pipeline. Input a valid name for your pipeline.

1. Configure Stage

Here, select your Jenkins CI account to take Build information. Prior to this step, you should have your Jenkins configured in Halyard under CI systems.


2. Bake Stage

We are targetting the package to Ubuntu server (16+), while the base image is baked with our app installed on it. The deb package file name should have been created with the naming convention: name_version-release_arch.

Here, select your package (.deb) name to create a baked image. The package is essentially the output of Jenkins Build. This stage will looke for a valid package within the Jenkins project directory.

Select your region ‘eastus‘ to deploy the application into. Remember the available regions here are
activated by hal command. Then, type your package name without any version information

3. Deploy Stage

The deployment server group in Spinnaker will create a VMScaleSet on Azure, the loadbalancer will route the traffic to the VM ScaleSet.


Add a Server group and configure every section in ‘Configure Deployment Cluster‘. Choose the account, select the region as ‘eastus‘, valid stack that may denote environment like dev/qa/stg. Detail may contain any text, but I chose to have ‘svrgroup’. Then choose the load balancer you had created in one of the above steps.

Choose the Virtual network and Subnet, where the VM ScaleSets will be created during CD pipeline execution.

You can choose the firewall that you had created earlier to control the traffic to the VM ScaleSet machines.


Choose the Instance type that best fits for your application. Here, we have gone with General purpose computer.

You can associate some tags that may be used as custom meta-data for your VMs, which can be used for filtering and other operations.


Save the configuration. And, you are done with your Pipeline configuration. You are now ready to execute your Pipeline and see how it goess.

Troubleshooting

01. Problem:

In Bake stage, package name was was mentioned as restapp_ (since our build output package was restapp_1.0_all.deb). The Baking process throw the error “Unable to find deployable artifact starting with [RestApp ] and ending with .deb in [] and [restapp_1.0_all.deb]”

Solution:

Make sure your deb package file name complies with the naming convention: name_version- release_arch. In the Bake stage, just mention the name part alone (donot include the underscore).

02. Problem:

The Baking process throw the error “==> azure-arm: E: Unable to locate package restapp”

Solution:

This is possibly that you have not configured your Deb path in Spinnaker. Update the rosco-local file with deb repo and redeploy with hal deploy apply command.

On halyard Pod/VM,

$ cat >> /home/spinnaker/.hal/default/profiles/rosco-local.yml <<-EOF
debianRepository: http://spindd.s3-website-us-west-2.amazonaws.com trusty main
azure:
  enabled: true
EOF
03. Problem:

The Baking process throw the error “==> azure-arm: E: Failed to fetch http://spindd.s3-website-us-west-2.amazonaws.com/dists/trusty/main/binary-amd64/Packages  404 Not Found
==> azure-arm: E: Some index files failed to download. They have been ignored, or old ones used instead.”

Solution:

If Deb repository is S3 bucket, you should enable web-hosting option and use the Endpoint url in the rosco-local.yml file. The endpoint URL becomes available once you enable Web-hosting.

04. Problem:

The Baking process throw the error “==> azure-arm: -> Additional Disk : skipping, managed disk was used…
Build ‘azure-arm’ errored: unexpected EOF”

Solution:

The problem occurs due to a defect in Packer version 1.4.2 as stated here
https://github.com/has hicorp/packer/issues/7816.
The fix is to update the Packer version to 1.4.4 which is packaged with Spinnaker 1.17. This is confirmed here https://github.com/spinnaker/rosco/pull/452. Either  update the Spinnaker to 1.17 or you can update the rosco.yml file with overridden docker image version which has the Packer 1.4.4 updated. See below how to update the docker version for  rosco.

$ cat > /home/spinnaker/.hal/default/service-settings <<-EOF
artifactId: gcr.io/spinnaker-marketplace/rosco:0.15.0-20191023172815
EOF
05. Problem:

Got error in the deployment stage…
“Exception ( Monitor Deploy )
Multiple error occurred: NotFound,NotFound. Please see details., Unexpected exception: null! Please log in into Azure Portal and manually delete any resource associated with the appxinazure-dev-fe-v000 server group such as storage accounts, internal load balancer, public IP and subnets”

Solution:

Spinnaker did not provide enough log information as to which resouce is NotFound. In tracing the resources everything we created, found that the items – VMUsername, and VMSshPublicKey were missing in Azure vault SpinHal-vault

After adding the SSH key and public-key, the deployment stages went successfully.

Refer the Azure CLI procedure again to create the Username and Public-key. In particular the commands,

saga@sagamc:~$ az keyvault secret set --name VMUsername --vault-name $VAULT_NAME --value $VMUSER
saga@sagamc:~$ az keyvault secret set --name VMSshPublicKey --vault-name $VAULT_NAME --value
"$VMUSER_KEY"
06. Problem:
Spinnaker Deployment stage is completed successfully. My loadbalancer is public one, but unable to access the app URL from Internet.
Solution:
Solution to this problem involves couple of Network related tracing.
a. Verify the public IP of the load-balancer is reachable by _ping_ command – ‘ping <public-ip-of-lb>‘.
b. Check the status of load-balancer is good as per the Azure portal: Monitor > Service Health > and then make sure that Subscription ID and Resource Type = Load Balancer are selected. If it fails, then your load balancer configuration is incorrect – fixt it.
c. Make sure the application is working on the VM (e.g. 172.16.0.5) and is lisening on the designated port.
curl http://localhost:8080/health’
c. Make sure the application URL using the VM’s IP is accessible from another VM (e.g. 172.16.0.6) on the same network – ‘curl http://172.16.0.5:8080/health
d. If the step above is working good, it could be NSG that is causing the inbound requests blocking requests to VMs. Even if you have a rule from ‘AzureLoadBalancer’ to ‘VirtualNetwork’ on port 8080, it may fail. Try adding a rule explicitly that would have the Source ‘Any’ and Destination ‘VirtualNetwork’ for port 8080 and have it ‘Allowed’
c. You can also verify from Internet about what all the ports are open to the public-ip of the load balancer using the command ‘nmap -v <public-ip> -Pn’.

References

https://www.spinnaker.io/setup/install/providers/azure/

https://www.spinnaker.io/guides/tutorials/codelabs/azure-vmss-source-to-prod/

 

Tagged , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *