Select Page
by

Kiran G

|
last updated on March 14, 2023
Share

Introduction

As the Spinnaker is evolving, new features are being incorporated and one of them is External Account Configuration which allows the configurations to be externalized via Git, Hashicorp Vault, databases, CredHub, etc. You may be asked if we can keep the cloud provider account details in a remote repository? If so, what if an account change happens in that remote configuration while the Spinnaker service is running? Will it take effect without service restart? What are the steps involved? What other use cases are possible with remote configuration?

This blog provides answers to the above questions. In Spinnaker world, the credentials configured through halyard using which Spinnaker deploys the applications into various Cloud Platforms are known as Accounts. Typically complete account configuration resides in hal config.  ‘hal deploy apply’ is a Spinnaker mantra for any account change to take effect. 

Broadly account can be of any cloud provider, artifactory, CI system, storage system, etc. When such an account is added or updated or deleted, ‘hal deploy apply’ makes Spinnaker understand the change. But this also means a restart of one or more affected Spinnaker services.  Spinnaker is built on the Spring framework and uses various Spring modules. One of such modules is Spring Cloud Configuration which allows dynamic external configuration using Config Server and Client implementation. A client application can be refreshed for reloading any account changes that the Config server is designed to capture. Spinnaker service need not be restarted for the account changes in external configuration. This refresh can be automatic or manual. As of Spinnaker 1.17.5, only Clouddriver service supports automatic refreshing of external accounts and only for Kubernetes and Cloud Foundry cloud providers. For manual triggering, /refresh endpoint on the service be http POSTed. 

External Account Configuration for Spinnaker

A file named spinnakerconfig.yml is a bootstrap configuration file that provides the location of the remote configuration so that Spinnaker services like clouddriver, igor fetch configuration from the remote location at startup. The following examples show the configuration with Git as a remote repository. Spinnaker uses profiles like <service>-local.yml for additional configuration and profile is loaded from the remote configuration, if present. Remote locations can be one or a mix of the following backends.

The next sections provide step by step instructions for the following dynamic configurations:

  • Account Configuration
  • Secrets Configuration
  • Secrets Encryption
  • Configuration Files

Using this dynamic account configuration feature, one or more cloud provider accounts(Kubernetes, Azure, AWS, Cloud Foundry, etc) and/or CI accounts and/or notification services can be configured in the external sources mentioned above.  Keeping all the secrets in plain text in hal config may not always align with organizational policies so using the dynamic configuration feature, it is possible to ship the secrets to external repositories. The secrets in the external repositories can also be encrypted and Spinnaker services decrypt them at runtime. Configuration files like Kubernetes config file, Google Cloud configuration file can also be kept in Git or other backends and it can be configured in Spinnaker to retrieve them at runtime.       

External Account Configuration for Spinnaker

This section explains how to configure the following accounts with Git as an external source. The same configuration can be extended to any other cloud provider accounts.

  • Kubernetes Account
  • AWS Account
  • Azure Account
  • Jenkins Master Account
  1. Create a Git repository for storing the external accounts

    Let the repository be private with the name spinnaker-config and the github account username be opsmx, so the git repository URL is https://github.com/OpsMx/enterprise-spinnaker.  

  2. Place spinnakerconfig.yml file under ~/.hal/default/profiles/

    The file spinnakerconfig.yml (or <service-name>config.yml, i.e., clouddriverconfig.yml, igorconfig.yml, etc) contains the external repository details using which Spinnaker services fetch the account configurations. Following spinnakerconfig.yml must be placed under ~/.hal/default/profiles/ in halyard pod or the machine in which halyard service is running. If a spinnaker service (say clouddriver pod) is configured to run without halyard, then the file is pushed under /opt/spinnaker/config/ in the respective spinnaker service pod at startup.  

3.Create a file ‘clouddriver-local.yml’ in the Git repository

The file clouddriver-local.yml contains all the cloud provider accounts that are to be externally configured. The following configurations are contained in this file:

Kubernetes Account

				
					spinnakerconfig.yml
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: github.com/opsmx/spinnaker-config.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo
				
			
				
					    kubernetes:
      enabled: true
      accounts:
      - name: my-k8s-v2
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces: []
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        kubeconfigFile: /home/opsmx/.hal/default/staging/dependencies/1031763043-config
        oAuthScopes: []
        onlySpinnakerManaged: false
				
			

Note: The kubeconfigFile is not configured in Git. External config file configuration is explained in the following sections. 

AWS Account

				
					  aws:
      enabled: true
      accounts:
      - name: my-aws-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '123456789012'
        regions:
        - name: us-west-2
        assumeRole: role/spinnakerManaged
        lifecycleHooks: []
      primaryAccount: my-aws-account
      bakeryDefaults:
        baseImages: []
        awsAccessKey: AKIAXXXXXXXXXXXZHKU5 
        awsSecretKey: YqfRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ
      accessKeyId: ALNMXXXXXXXXXXXZHKT8
      secretAccessKey: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl4dK
      defaultKeyPairTemplate: '{{name}}-keypair'
      defaultRegions:
      - name: us-west-2
      defaults:
        iamRole: BaseIAMRole
				
			

Azure Account

				
					 azure:
      enabled: true
      accounts:
      - name: my-azure-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        clientId: 921f7add-XXXX-XXXX-XXXX-c1a695s08755
        appKey: 23373f24-XXXX-XXXX-XXXX-6b0b5113f5a0
        tenantId: d1ea7473-XXXX-XXXX-XXXX-ad8041071ee6
        subscriptionId: 959c83bf-XXXX-XXXX-XXXX-2067be19692e
        defaultResourceGroup: RG-OpsMx
        defaultKeyVault: SpinHal-OpsMx-Vault
        packerResourceGroup: RG-OpsMx
        regions:
        - westus
        - eastus
        useSshPublicKey: 'true'
      primaryAccount: my-azure-account
      bakeryDefaults:
        templateFile: azure-linux.json
        baseImages: []
				
			

For simplicity, only one account per cloud provider is configured but they can be more than one. Note that any accounts of the above three cloud provide configured in hal config are overridden by the above configuration.

4.Create a file ‘igor-local.yml’ in the Git repository

We configure the Jenkins Master account in igor-local.yml and place it directly under the Git repository created in step 1. The following configuration is contained in this file:

				
					 ci:
    jenkins:
      enabled: true
      masters:
      - name: opsmx-jenkins-master
        permissions: {}
        address: http://jenkins.opsmx.com
        username: opsmx
        password: mypassword
				
			

5. Run ‘hal deploy apply’

At this point, there are two files – clouddriver-local.yml and igor-local.yml – in the Git repository, and Git repository details are configured in ~/.hal/default/profiles/spinnakerconfig.yml. Upon running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into all spinnaker service pods under /opt/spinnaker/config/ and the pods fetch the respective -local.yml files into their /tmp/config-repo/ directory and load the accounts. In our scenario, clouddriver and igor pods load the accounts from clouddriver-local.yml and igor-local.yml files respectively.  

Secrets Configuration for External Accounts

The following procedure details the secrets configuration using Git. The rest of the configuration lies with the halyard. Two scenarios are explained here: 1. AWS access key id and secret key and 2. Jenkins Master address, username, and password.  

 

  1. Configure hal config with Spring property placeholders

    AWS Account Configuration in hal config

    Configure AWS Account in hal config with spring property placeholders for AWS Access Key Id and Secret Key which get replaced with the actual values at run time by the clouddriver service. After configuring, the AWS part of the hal config looks like the following:

				
					aws:
      enabled: true
      accounts:
      - name: my-aws-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '732813442182'
        regions:
        - name: us-west-2
        assumeRole: role/spinnakerManaged
        lifecycleHooks: []
      primaryAccount: my-aws-account
      bakeryDefaults:
        baseImages: []
      accessKeyId: ${aws.access-key}
      secretAccessKey: ${aws.secret-key}
      defaultKeyPairTemplate: '{{name}}-keypair'
      defaultRegions:
      - name: us-west-2
      defaults:
        iamRole: BaseIAMRole 
				
			

Jenkins Master Account Configuration in hal config

Similarly configure hal config for Jenkins Master Account with spring property placeholders for Jenkins Master address, username, and password as shown below:

				
					 ci:
    jenkins:
      enabled: true
      masters:
      - name: opsmx-jenkins-master
        permissions: {}
        address: ${jenkins.address}
        username: ${jenkins.username}
        password: ${jenkins.password}
				
			

2.Create a Git repository for storing the secrets

Let the repository be private with name secrets-config and the GitHub account username be opsmx, so the git repository URL is https://github.com/OpsMx/enterprise-spinnaker/tree/master/charts/oes/templates/secrets.

3.Create a clouddriver-local.yml file in the Git repository

The file clouddriver-local.yml contains the actual values for the AWS Access Key and Secret Key:

				
					aws:
    access-key: ALIAXXXXXXXXXXXZHKU5
    secret-key: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ
				
			

4. Create an igor-local.yml file in the Git repository

The file igor-local.yml contains the actual values for the Jenkins Master Account’s address, username, and password:

				
					jenkins:
    address: http://jenkins.opsmx.com
    username: opsmx
    password: mypassword
				
			

5.Enable external configuration through spinnakerconfig.yml

Place the file spinnakerconfig.yml under ~/.hal/default/profiles/ in the halyard pod or the machine in which the halyard service is running with the following configuration.
				
					spinnakerconfig.yml
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: github.com/opsmx/secrets-config.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo
				
			

6. Run ‘hal deploy apply’

Now all the configuration is ready and by running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into clouddriver and igor pods which fetch the secrets from Git and replace the spring property placeholders at runtime.

Secrets Encryption in Spinnaker

Please refer to this blog to know how to externalize encrypted secrets: Managing Secrets in Spinnaker – Encryption using Symmetric Key

Dynamic External Account Configuration File for Spinnaker

The following steps explain how to configure a kubeconfig file that is checked-in a git repository.

  1. Create a Git repository for storing the config files

    Let the repository be private with name config-files and the github account username be opsmx, so the git repository URL is https://github.com/OpsMx/standard-gitops-repo/blob/master/config

  2. Create a file kubeconfig.yml in the Git repository

  3. Create kubeconfig.yml which provides cluster access for organizing Kubernetes accounts, in the Git repository created in step 1.

  4. Refer kubeconfig.yml in clouddriver-local.yml

    Create a clouddriver-local.yml file in the Git repository file which contains the Kubernetes account details including the path to the kubeconfig file. Since the clouddriver service is backed by Spring Cloud’s config server, by configuring appropriately, it fetches the kubeconfig file from the Git repository.  

				
					clouddriver-local.yml
    kubernetes:
      enabled: true
      accounts:
      - name: my-k8s-v2
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces: []
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        kubeconfigFile: configserver:kubeconfig.yml
        oAuthScopes: []
        onlySpinnakerManaged: false
				
			

4. Place spinnakerconfig.yml file under ~/.hal/default/profiles/

Place the file spinnakerconfig.yml under ~/.hal/default/profiles/ in the halyard pod or the machine in which the halyard service is running with the following configuration.

				
					spinnakerconfig.yml
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: github.com/opsmx/config-files.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo
				
			

5. Run ‘hal deploy apply’

By running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into the clouddriver pod which loads the Kubernetes account and kubeconfig.yml from Git.

Note – Currently, dynamic accounts only work with Kubernetes and not AWS.


If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us

0 Comments

Submit a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.