Introduction
In the blog, we will explain how SREs can accurately verify the risk of their software in CI/CD pipeline by integrating Autopilot with Datadog monitoring solutions.
OpsMx Autopilot is a machine learning (ML) and natural language processing tool that analyzes the data for you automatically so you can quickly and accurately decide whether an update should be moved forward in the pipeline. Autopilot helps you to stay a step ahead of the competition by automating the decision-making process and assessing risk before deployment. Autopilot is a verification module, which is a part of the larger OpsMx platform for continuous delivery built on top of Spinnaker. It follows API based architecture, which is extremely easy to extend and integrate with any DevOps tool chain in your organization.
Growing importance of monitoring a CI/CD pipeline
With the introduction of automation tools into the software delivery process, things are moving fast. When we try to release fast, things break. So it becomes crucial for SREs to monitor for logs and metrics to detect and troubleshoot breakdowns.
The most important part of monitoring is analyzing whether a newly developed software change is fit for the production system. Logs and Metrics both serve different purposes and have to be analyzed differently. Logs are necessary for debugging and auditing a software delivery pipeline. Whereas metrics are used for monitoring and alerting. They help trigger notifications almost instantaneously.
These logs and metrics can be analyzed by using platforms like Datadog. In recent years, the importance of such monitoring tools has increased because CI/CD pipeline adoption has increased exponentially. With releases being released into production frequently, SREs are always on their toes to ensure zero downtime and avoid a breach of services because of buggy software.
But real-time notifications and error detections will not solve the organisation problem to deliver good software. There needs to be something that can understand the risk of a release before we put it into production to take preventive action.
Challenges with manual verification in Datadog
SREs perform manual analysis to troubleshoot the problems that are generated from Datadog. Some issues may be known and some unknown. Post analysis, a SRE has to take measures to apply a fix. This process might delay the next pending release and affect customers.
-
Inaccuracies
A typical error triggers a series of steps that the SREs need to follow. It starts by analyzing metrics from the rollout strategy and locating metrics that are below or above a certain threshold. Next, they pull logs to further drill down on a granular level to pinpoint the error element. When the scope is modest and SREs have only a few pipelines, they can easily manage risks by manually monitoring their dashboards. But where an organisation runs hundreds of pipelines, manual troubleshooting may take ages and troubleshooting can be inaccurate because of short timelines.
-
Increased triage time
When organizations are operating at a scale, the complexity of managing logs and metrics and performing triage is nearly impossible, even for an expert team of SREs. Deployment pipelines may throw out thousands of exceptions a day. It is impossible for that team of expert SREs to keep up with the vast amount of error reports.
-
Dependencies on multiple platforms and tools
From the thousands of exceptions generated from multiple applications, hundreds of them may require manual analysis and judgement. A few scripts for known issues may lead to temporary success, but it is never a long-term solution. They are also time-consuming and hard to maintain. SREs are costly resources and should not be occupied doing risk assessments all the time. Any error that goes unnoticed is a very costly mistake.
It is very clear that organizations need to look further than just dashboards and reporting. This is where OpsMx Autopilot fills the gap.
What is Autopilot, and how does it overcome the challenges?
Autopilot understands application and system logs, performs a risk assessment, and can control the CI/CD pipeline through an approval or rejection decision. It can also detect unsuspecting error logs that may go past a human reviewer or any logic based filters that might cause your deployment to fail in production.
Autopilot is an intelligence layer for any Continuous Delivery Platform and Log analyzer. We can add this to a Datadog platform to perform advanced data analytics and predictive modeling to expedite business decisions. With inbuilt features of verification gates, Autopilot can perform risk assessment scores on the quality of deployment. These assessments are done in a matter of seconds and provide an in-depth view of any arising problems in the update. Autopilot can not only be integrated with a machine log analyzer tool but also with Jenkins or any other CICD tools, giving it a 360 view of the overall process. With a loop-back mechanism, the risk assessment improves over time and reduces the need for an SRE to monitor and analyze every report that comes his or her way.
If you are running a Datadog ecosystem, Autopilot will integrate with it to automate risk assessment, thus saving a tremendous amount of time. Besides improving the accuracy of promotion decisions, Autopilot can also automatically approve or reject deployments in your CI/CD Pipeline. This is possible by providing benchmark scores that will auto adjust over time to improve risk scores. In short, Autopilot simplifies and improves time-consuming and error-prone processes.
Read the next blog on how autopilot achieves the realtime risk assessment and what more you can expect out of Autopilot.
Get Started With OpsMx Autopilot
Integrate OpsMx Autopilot with Datadog for Realtime Risk assessment in 4 easy Steps
Getting started with OpsMx Autopilot is as easy as signing into a social media platform. The API key enables Autopilot to fetch data from your Datadog servers automatically. Once linked you can managed all your releases from the Autopilot dashboard.
The integration can be achieved in 4 easy steps :
- Generate an API access key form your Datadog portal
- Link your existing monitoring platform from our list of integrations
- Collect our existing logs and metrics to perform Risk assessments or define some of your own
- View and Manage Risk assessments and approvals for a unified autopilot Dashboard.
0 Comments
Trackbacks/Pingbacks