Introduction
As the Spinnaker is evolving, new features are being incorporated and one of them is External Account Configuration which allows the configurations to be externalized via Git, Hashicorp Vault, databases, CredHub, etc. You may be asked if we can keep the cloud provider account details in a remote repository? If so, what if an account change happens in that remote configuration while the Spinnaker service is running? Will it take effect without service restart? What are the steps involved? What other use cases are possible with remote configuration?
This blog provides answers to the above questions. In Spinnaker world, the credentials configured through halyard using which Spinnaker deploys the applications into various Cloud Platforms are known as Accounts. Typically complete account configuration resides in hal config. ‘hal deploy apply’ is a Spinnaker mantra for any account change to take effect.
Broadly account can be of any cloud provider, artifactory, CI system, storage system, etc. When such an account is added or updated or deleted, ‘hal deploy apply’ makes Spinnaker understand the change. But this also means a restart of one or more affected Spinnaker services. Spinnaker is built on the Spring framework and uses various Spring modules. One of such modules is Spring Cloud Configuration which allows dynamic external configuration using Config Server and Client implementation. A client application can be refreshed for reloading any account changes that the Config server is designed to capture. Spinnaker service need not be restarted for the account changes in external configuration. This refresh can be automatic or manual. As of Spinnaker 1.17.5, only Clouddriver service supports automatic refreshing of external accounts and only for Kubernetes and Cloud Foundry cloud providers. For manual triggering, /refresh endpoint on the service be http POSTed.
External Account Configuration for Spinnaker
A file named spinnakerconfig.yml is a bootstrap configuration file that provides the location of the remote configuration so that Spinnaker services like clouddriver, igor fetch configuration from the remote location at startup. The following examples show the configuration with Git as a remote repository. Spinnaker uses profiles like <service>-local.yml for additional configuration and profile is loaded from the remote configuration, if present. Remote locations can be one or a mix of the following backends.
- Git
- Hashicorp Vault
- JDBC
- CredHub
- Local File System
The next sections provide step by step instructions for the following dynamic configurations:
- Account Configuration
- Secrets Configuration
- Secrets Encryption
- Configuration Files
Using this dynamic account configuration feature, one or more cloud provider accounts(Kubernetes, Azure, AWS, Cloud Foundry, etc) and/or CI accounts and/or notification services can be configured in the external sources mentioned above. Keeping all the secrets in plain text in hal config may not always align with organizational policies so using the dynamic configuration feature, it is possible to ship the secrets to external repositories. The secrets in the external repositories can also be encrypted and Spinnaker services decrypt them at runtime. Configuration files like Kubernetes config file, Google Cloud configuration file can also be kept in Git or other backends and it can be configured in Spinnaker to retrieve them at runtime.
External Account Configuration for Spinnaker
This section explains how to configure the following accounts with Git as an external source. The same configuration can be extended to any other cloud provider accounts.
- Kubernetes Account
- AWS Account
- Azure Account
- Jenkins Master Account
Create a Git repository for storing the external accounts
Let the repository be private with the name spinnaker-config and the github account username be opsmx, so the git repository URL is https://github.com/OpsMx/enterprise-spinnaker.
Place spinnakerconfig.yml file under ~/.hal/default/profiles/
The file spinnakerconfig.yml (or <service-name>config.yml, i.e., clouddriverconfig.yml, igorconfig.yml, etc) contains the external repository details using which Spinnaker services fetch the account configurations. Following spinnakerconfig.yml must be placed under ~/.hal/default/profiles/ in halyard pod or the machine in which halyard service is running. If a spinnaker service (say clouddriver pod) is configured to run without halyard, then the file is pushed under /opt/spinnaker/config/ in the respective spinnaker service pod at startup.
3.Create a file ‘clouddriver-local.yml’ in the Git repository
The file clouddriver-local.yml contains all the cloud provider accounts that are to be externally configured. The following configurations are contained in this file:
Kubernetes Account
spinnakerconfig.yml
spring:
profiles:
include: git
cloud:
config:
server:
git:
uri: github.com/opsmx/spinnaker-config.git
username: opsmx
password: xxxxxxxx
basedir: /tmp/config-repo
kubernetes:
enabled: true
accounts:
- name: my-k8s-v2
requiredGroupMembership: []
providerVersion: V2
permissions: {}
dockerRegistries: []
configureImagePullSecrets: true
cacheThreads: 1
namespaces: []
omitNamespaces: []
kinds: []
omitKinds: []
customResources: []
cachingPolicies: []
kubeconfigFile: /home/opsmx/.hal/default/staging/dependencies/1031763043-config
oAuthScopes: []
onlySpinnakerManaged: false
Note: The kubeconfigFile is not configured in Git. External config file configuration is explained in the following sections.
AWS Account
aws:
enabled: true
accounts:
- name: my-aws-account
requiredGroupMembership: []
providerVersion: V1
permissions: {}
accountId: '123456789012'
regions:
- name: us-west-2
assumeRole: role/spinnakerManaged
lifecycleHooks: []
primaryAccount: my-aws-account
bakeryDefaults:
baseImages: []
awsAccessKey: AKIAXXXXXXXXXXXZHKU5
awsSecretKey: YqfRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ
accessKeyId: ALNMXXXXXXXXXXXZHKT8
secretAccessKey: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl4dK
defaultKeyPairTemplate: '{{name}}-keypair'
defaultRegions:
- name: us-west-2
defaults:
iamRole: BaseIAMRole
Azure Account
azure:
enabled: true
accounts:
- name: my-azure-account
requiredGroupMembership: []
providerVersion: V1
permissions: {}
clientId: 921f7add-XXXX-XXXX-XXXX-c1a695s08755
appKey: 23373f24-XXXX-XXXX-XXXX-6b0b5113f5a0
tenantId: d1ea7473-XXXX-XXXX-XXXX-ad8041071ee6
subscriptionId: 959c83bf-XXXX-XXXX-XXXX-2067be19692e
defaultResourceGroup: RG-OpsMx
defaultKeyVault: SpinHal-OpsMx-Vault
packerResourceGroup: RG-OpsMx
regions:
- westus
- eastus
useSshPublicKey: 'true'
primaryAccount: my-azure-account
bakeryDefaults:
templateFile: azure-linux.json
baseImages: []
For simplicity, only one account per cloud provider is configured but they can be more than one. Note that any accounts of the above three cloud provide configured in hal config are overridden by the above configuration.
4.Create a file ‘igor-local.yml’ in the Git repository
We configure the Jenkins Master account in igor-local.yml and place it directly under the Git repository created in step 1. The following configuration is contained in this file:
ci:
jenkins:
enabled: true
masters:
- name: opsmx-jenkins-master
permissions: {}
address: http://jenkins.opsmx.com
username: opsmx
password: mypassword
5. Run ‘hal deploy apply’
At this point, there are two files – clouddriver-local.yml and igor-local.yml – in the Git repository, and Git repository details are configured in ~/.hal/default/profiles/spinnakerconfig.yml. Upon running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into all spinnaker service pods under /opt/spinnaker/config/ and the pods fetch the respective -local.yml files into their /tmp/config-repo/ directory and load the accounts. In our scenario, clouddriver and igor pods load the accounts from clouddriver-local.yml and igor-local.yml files respectively.
Secrets Configuration for External Accounts
The following procedure details the secrets configuration using Git. The rest of the configuration lies with the halyard. Two scenarios are explained here: 1. AWS access key id and secret key and 2. Jenkins Master address, username, and password.
Configure hal config with Spring property placeholders
AWS Account Configuration in hal config
Configure AWS Account in hal config with spring property placeholders for AWS Access Key Id and Secret Key which get replaced with the actual values at run time by the clouddriver service. After configuring, the AWS part of the hal config looks like the following:
aws:
enabled: true
accounts:
- name: my-aws-account
requiredGroupMembership: []
providerVersion: V1
permissions: {}
accountId: '732813442182'
regions:
- name: us-west-2
assumeRole: role/spinnakerManaged
lifecycleHooks: []
primaryAccount: my-aws-account
bakeryDefaults:
baseImages: []
accessKeyId: ${aws.access-key}
secretAccessKey: ${aws.secret-key}
defaultKeyPairTemplate: '{{name}}-keypair'
defaultRegions:
- name: us-west-2
defaults:
iamRole: BaseIAMRole
Jenkins Master Account Configuration in hal config
Similarly configure hal config for Jenkins Master Account with spring property placeholders for Jenkins Master address, username, and password as shown below:
ci:
jenkins:
enabled: true
masters:
- name: opsmx-jenkins-master
permissions: {}
address: ${jenkins.address}
username: ${jenkins.username}
password: ${jenkins.password}
2.Create a Git repository for storing the secrets
Let the repository be private with name secrets-config and the GitHub account username be opsmx, so the git repository URL is https://github.com/OpsMx/enterprise-spinnaker/tree/master/charts/oes/templates/secrets.
3.Create a clouddriver-local.yml file in the Git repository
The file clouddriver-local.yml contains the actual values for the AWS Access Key and Secret Key:
aws:
access-key: ALIAXXXXXXXXXXXZHKU5
secret-key: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ
4. Create an igor-local.yml file in the Git repository
The file igor-local.yml contains the actual values for the Jenkins Master Account’s address, username, and password:
jenkins:
address: http://jenkins.opsmx.com
username: opsmx
password: mypassword
5.Enable external configuration through spinnakerconfig.yml
spinnakerconfig.yml
spring:
profiles:
include: git
cloud:
config:
server:
git:
uri: github.com/opsmx/secrets-config.git
username: opsmx
password: xxxxxxxx
basedir: /tmp/config-repo
6. Run ‘hal deploy apply’
Now all the configuration is ready and by running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into clouddriver and igor pods which fetch the secrets from Git and replace the spring property placeholders at runtime.
Secrets Encryption in Spinnaker
Please refer to this blog to know how to externalize encrypted secrets: Managing Secrets in Spinnaker – Encryption using Symmetric Key
Dynamic External Account Configuration File for Spinnaker
The following steps explain how to configure a kubeconfig file that is checked-in a git repository.
Create a Git repository for storing the config files
Let the repository be private with name config-files and the github account username be opsmx, so the git repository URL is https://github.com/OpsMx/standard-gitops-repo/blob/master/config
Create a file kubeconfig.yml in the Git repository
Create kubeconfig.yml which provides cluster access for organizing Kubernetes accounts, in the Git repository created in step 1.
Refer kubeconfig.yml in clouddriver-local.yml
Create a clouddriver-local.yml file in the Git repository file which contains the Kubernetes account details including the path to the kubeconfig file. Since the clouddriver service is backed by Spring Cloud’s config server, by configuring appropriately, it fetches the kubeconfig file from the Git repository.
clouddriver-local.yml
kubernetes:
enabled: true
accounts:
- name: my-k8s-v2
requiredGroupMembership: []
providerVersion: V2
permissions: {}
dockerRegistries: []
configureImagePullSecrets: true
cacheThreads: 1
namespaces: []
omitNamespaces: []
kinds: []
omitKinds: []
customResources: []
cachingPolicies: []
kubeconfigFile: configserver:kubeconfig.yml
oAuthScopes: []
onlySpinnakerManaged: false
4. Place spinnakerconfig.yml file under ~/.hal/default/profiles/
Place the file spinnakerconfig.yml under ~/.hal/default/profiles/ in the halyard pod or the machine in which the halyard service is running with the following configuration.
spinnakerconfig.yml
spring:
profiles:
include: git
cloud:
config:
server:
git:
uri: github.com/opsmx/config-files.git
username: opsmx
password: xxxxxxxx
basedir: /tmp/config-repo
5. Run ‘hal deploy apply’
By running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into the clouddriver pod which loads the Kubernetes account and kubeconfig.yml from Git.
Note – Currently, dynamic accounts only work with Kubernetes and not AWS.
If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.
OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us
0 Comments