Auto cancellation

Automatically cancel complex Terraform applies when alarms fire

Overview

One common issue when managing changes to Kubernetes infrastructure is due to the very long running nature of the operations, cluster upgrades can take hours for large node counts, there is plenty that can go wrong and immense wasted man-hours babying your infrastructure automation ensuring that does not happen.

Plural helps solve this by polling the monitors you likely have set up to ensure infrastructure health in tools like Datadog or NewRelic, and automatically cancelling your IaC when they fire. Due to our close management of the commands themselves, we'll gracefully shut them down, ensuring things like annoying state locks are cleaned up and no resources are left dangling. We're basically trying to automate one of the most boring but labor-intensive parts of your DevOps workflow.

Setting this up is really simple, you'll need to create an ObservabilityProvider resource and then set a list of observableMonitors on your stack.

Create an ObservabilityProvider

To do this in one swoop for datadog, create resources like:

yaml

apiVersion: v1
kind: Secret
metadata:
  name: datadog
  namespace: stacks
stringData:
  apiKey: YOUR_API_KEY
  appKey: YOUR_APP_KEY
---
apiVersion: deployments.plural.sh/v1alpha1
kind: ObservabilityProvider
metadata:
  name: datadog
spec:
  type: DATADOG
  name: datadog
  credentials:
    datadog:
      name: datadog
      namespace: stacks

Info:

You can also create this in the UI to avoid the secret creation, and reference it by name without the credentials block

Add `observableMetrics` to your Stack

An example setup for this is here: