- Spinnaker 1.17.6 (However the same can work on new versions of Spinnaker)
- Hashicorp Vault as external kubeconfig files store
- Halyard service’s deployment profile is the default (i.e $HOME/.hal/default directory)
Spinnaker loads Kubernetes provider accounts using the clouddriver service. In most cases, all of the Spinnaker services’ accounts are stored in halconfig file. If we configure Spinnaker’s inclusive ‘Spring Cloud Config server’ with Git external store, it can read the accounts from a Git repository. In any case, the kubeconfig file path is inferred as an absolute path pertaining to Clouddriver Pod.
Either way, we want to store the kubeconfig files in a Vault store. The kubeconfig files of the accounts are stored under a dedicated path in a backend engine. These kubeconfig files are pulled to Clouddriver Pod using a periodic/Cron script. This pull is achieved using a Sidecar container within Clouddriver Pod.
The kubeconfig file path on the Cloulddriver Pod should be the same as in the configuration of the Clouddriver accounts.You can change the Vault backend, secret path, and other values as per your need.
- Vault administrator creates a Token for Spinnaker
- Store the Kubernetes Accounts’ kubeconfig files in Vault
- Update Clouddriver Pod with Sidecar container to pull Kubeconfig files
- Verify the Kubernetes Accounts’ kubeconfig files downloaded correctly
Detailed Procedure
1. Vault administrator creates a Token for Spinnaker
2. Store the Kubernetes Accounts’ kubeconfig files in Vault
vault kv put spinnaker/k8configs/my-account kubeconfig=@my-account.yml
3. Update Clouddriver Pod with Sidecar container to pull Kubeconfig files
- Checks for any new account
vault kv list spinnaker/k8configs
- Downloads the new account’s kubeconfig file
vault kv get -field=kubeconfig spinnaker/k8configs/my-account > my-account.yml
#!/bin/bash
#This script is executed in Clouddriver or its Sidecard container to download the kubeconfig files
#Make sure vault cli is loaded on the Container and is accessible by the scripts
#Ensure the VAULT_ADDR and VAULT_TOKEN is set in the SHELL environment
#Set these variables to your choice
export VK8S_PATH='spinnaker/k8configs' #Prefix V for Vault
export LK8S_PATH='/tmp/k8configs' #Prefix L for Local
export R_FILE=/tmp/remote.txt
export L_FILE=/tmp/local.txt
export NEWKEYS=/tmp/newkeys.txt
#Make sure vault CLI is available
command -v vault > /dev/null 2>&1
if [ $? -ne 0 ]; then
echo "Vault CLI is not available, exiting..."
exit
fi
#Create local config path if not exists
[ ! -d $LK8S_PATH ] && (mkdir -p $LK8S_PATH; echo Created k8sconfig directory) #|| echo Content mkdir -p $LK8S_PATH
#Set-1
vault kv list $VK8S_PATH | egrep -iv 'Keys|----' > $R_FILE
#Set-2
ls -1 $LK8S_PATH | sed s/\.yml$// > $L_FILE
#New in Vault
comm -23 $R_FILE $L_FILE > $NEWKEYS
if [ ! -s $NEWKEYS ]; then
echo "There are no new kubeconfigs to download"
exit
fi
#Download new kubeconfig file from Vault
for i in `cat $NEWKEYS`; do
echo -en "Fetching kubeconfig file of Account : $i"
vault kv get -field=kubeconfig $VK8S_PATH/$i > $LK8S_PATH/$i.yml
if [ $? -eq 0 ]; then
echo " [success]"
echo " File: $LK8S_PATH/$i.yml"
else
echo " [failed with return code $?]"
fi
done
echo "Completed Kubeconfig downlod syncing"
The Vault address and its token are stored as secret
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: vaultsec
namespace: spinnaker
data:
#Replace the vaule here http://vault-host as vaultaddress, and token as encrypted using base64
vaultaddr: abcovLzE3Mi40Mi40Mi4xMTE6ODIwMA==
vaulttoken: abca054QXhQbW01T3hDUzVNaU1RbWl1Qmc=
Configuring the Sidecar container with the script:
kind: Deployment
metadata:
labels:
spec:
template:
metadata:
spec:
containers:
- name: clouddriver
volumeMounts:
- mountPath: /tmp/k8configs
name: vvault
- name: vault-c
image: alpine:latest
command: ["/bin/sh", " -c"]
args:
- |
while true; do sh /tmp/k8configs-sync.sh; sleep 30m; done
env:
- name: VAULT_ADDR
valueFrom:
secretKeyRef:
key: vaultaddr
name: vaultsec
- name: VAULT_TOKEN
valueFrom:
secretKeyRef:
key: vaulttoken
name: vaultsec
volumeMounts:
- mountPath: /tmp/k8configs
name: vvault
- mountPath: /tmp/k8configs-sync.sh
name: vcm-vault
subPath: k8configs-sync.sh
volumes:
- emptyDir: {}
name: vvault
- name: vcm-vault
configMap:
defaultMode: 420
name: vault-k8s
- A shared directory volume of type ‘EmptyDir’ is created and mounted to both containers under the path ‘/tmp/k8configs’. The ‘EmptyDir’ type creates an empty directory on the Host machine before the containers are started, and then shared between the colocated containers as long as the volume is referenced
- The ConfigMap is mounted to the sidecar container as a script in the path /tmp/k8configs-sync.sh, which gets executed every 30 minutes to download any new kubeconfig files.
4. Verify the Kubernetes Accounts’ kubeconfig files downloaded correctly
- Go to Clouddriver’s sidecar container and check if the kubeconfig files are downloaded.
- Also, go to Clouddriver’s main container and verify if the kubeconfig file is available in the same path as in the Account’s kubeconfig file reference
- We can login to Spinnaker and check the URL http:///credentials to see available accounts.
About OpsMx
Founded with the vision of “delivering software without human intervention,” OpsMx enables customers to transform and automate their software delivery processes. OpsMx builds on open-source Spinnaker and Argo with services and software that helps DevOps teams SHIP BETTER SOFTWARE FASTER.
0 Comments