Spinnaker – Blue/Green strategy with external versioning and Kubernetes Deployment object

From this blog you can explore options to implement externally versioned or managed blue/green strategy for application deployment using spinnaker pipeline.

Introduction and Scope

Blue-green deployment strategy for production environments reduces downtime as well as risk by running two identical deployments called Blue and Green. At any given time, there is only one live deployment serving complete production traffic.

This technique enables development teams to achieve continuous deployment with final stage of testing on production ready newer versions (say Blue) before making it live and once passed, then complete incoming traffic switched using a Load Balancer from existing deployment (say Green) to Blue. This strategy supports:

    1. Faster rollouts for business to release new features on demand.
    2. Easier rollback to older version of deployment in case of something unexpected identified in newer deployment.
    3. Simultaneous rollout of deployments with multi-service dependencies.

Spinnaker natively supports Blue/Green deployment strategy with Kubernetes ReplicaSet object and has a spinnaker managed versioning system for rolling forward and roll back. However, Spinnaker (till 1.19.x) does not support Blue/Green natively for Deployment objects and in certain use cases organizations want to maintain versions out of Spinnaker. So, in this blog we will focus on following requirements:

    1. Use Kubernetes Deployment object for blue/green strategy.
    2. Passing the version of blue deployment from externally managed source as pipeline parameter.
    3. Pipeline designs to achieve above requirement and possibility to extend it as pipeline template for various teams.

Possible approaches to design Solution

In my scenario I will design the sample spinnaker pipelines using “nginx” image for simplicity. And here we are going to use pipeline expressions feature on Spinnaker, to develop this solution. This solution uses Kubernetes service object as Load Balancer to switch traffic from blue to green or vice versa, based on matching selector labels in service object to that of spec.template.metadata.labels in deployment object.

Implementation

  • To manage versions, use of parameterized deployment manifest such as given below:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: 'nginx-deployment-${parameters.release}'
spec:
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: nginx
        release: '${parameters.release}'
    spec:
      containers:
        - image: 'nginx:1.7.9'
          name: nginx
          ports:
            - containerPort: 80
    • Assuming service object pre-exist with “release” as one of the selector labels such as given below, and holding the “release” value same as mentioned in first deployment release:
apiVersion: v1
kind: Service
metadata:
    name: nginx-svc
    namespace: default
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
    release: '${parameters.release}'

Step By Step Guidelines to Design Pipeline

  • Pipeline – 1:
    • Pipeline should consist of all required parameters for release and 2 stages
    • Deploy Stage for Blue Deployment

    • Patch Stage for service to switch traffic on Blue deployment.
  • Pipeline – 2:
    • In this scenario pipeline should consist of 3 stages
    • Patch Stage for service to share traffic between existing green and upcoming blue.

 

 

 

    • Deploy Stage for Blue deployment.
    • Patch Stage for service to switch traffic completely on Blue deployment.
  • With both the approaches, rollout and rollback can be achieved with parameters of given choice. While going for the second design, we can achieve a blue/green strategy with smooth transitioning of traffic as well as non-live deployment to achieve quick roll back.
  • However, there is a challenge in specific conditions, if more than 1 release (including green) version exists in environment. Then traffic will be bound to share in all of them after the first stage. This will pose an issue in the production environment.
  • The above discussed pipelines can be extended to use git/bitbucket based Kubernetes artifacts instead of text based. And further can be parameterized to use it as a pipeline template for different teams.

Possibilities of Improvement in given approaches

  • The above suggested approaches does not clean up any of those existing non-live/older deployments. The older deployments can be removed by executing either of the below two options
    • By deleting Deployments from UI (Infrastructure tab > Clusters sub-tab)
    • By creating a separate pipeline, which contains delete manifest stage and release parameter.
  • This approach also assumes a robust external versioning system for release management.

Conclusion

The deployment strategies play vital role in achieving faster continuous delivery (CD). Blue/green strategy is one of the prominent production deployment strategies, used by organizations. The above steps help in understanding as well as achieving blue/green strategy in containerized production environment using Kubernetes objects and spinnaker pipelines.

Follow OpsMx blog for more such Enterprise grade solutions for Spinnaker 🙂

Tagged , ,

Leave a Reply

Your email address will not be published. Required fields are marked *