Overview

Alerting is supported in the DC/OS Monitoring service via Prometheus Alertmanager. The DC/OS Monitoring service is automatically configured for Prometheus to talk to Alertmanager. See the Alertmanager documentation for instructions on how to configure and turn on Alertmanager.

This page explains how to load Prometheus alert rules used for alerting. See the Prometheus documentation for instructions on how to define these rules.

Automatically loading Prometheus alert rules

The DC/OS Monitoring service can be configured to automatically load Prometheus alert rules from a Git repository.

Save Prometheus alert rules

You should save your Prometheus alert rules (YAML format) in a Git repository. Assume that the repository is https://github.com/company/alert-rules.

You can put the alert rules in a folder in the repository. For instance, https://github.com/company/alert-rules/production.

{
  "prometheus": {
    "alert_rules_repository": {
      "config_repository": {
        "url": "https://github.com/company/alert-rules",
        "path": "/production"
      }
    }
  }
}

IMPORTANT: Make sure that each of the alert rule files has suffix .yml and is a direct child of the folder (that is not nested under another folder).

The Git repository can be either public or private. If the Git repository is private, you will need to configure the credentials to access the Git repository (see below).

NOTE: Mesosphere maintains a Git repository with Prometheus alerting rule configurations at https://github.com/dcos/prometheus-alert-rules. If you would like to use these Prometheus alerting rules, you should set prometheus.alert_rules_repository.config_repository.url to https://github.com/dcos/prometheus-alert-rules and prometheus.alert_rules_repository.config_repository.path to /rules. As it is a public repository, there is no need to set up the credentials.

{
  "prometheus": {
    "alert_rules_repository": {
      "config_repository": {
        "url": "https://github.com/dcos/prometheus-alert-rules",
        "path": "/rules"
      }
    }
  }
}

Create secrets for Git repository credentials

If the Git repository containing the alert rules is private, you will need to configure the secrets first. Currently, the following Auth types are supported.

IMPORTANT: If the Git repository contains submodules, all the submodules must use the same Auth type as what is used for the repository. For example, if HTTP Auth is configured for the Git repository, all of the submodules in the repository's .gitmodules file must use the https URL scheme.

HTTP Auth

dcos security secrets create --value=<GITHUB_USERNAME> githubusername-secret
dcos security secrets create --value=<GITHUB_PASSWORD> githubpassword-secret

For GitHub, the password can be the API token.

Create a custom option file (options.json) like the following. You may omit the credentials section if the Git repository is public.

{
  "prometheus": {
    "alert_rules_repository": {
      "url": "https://github.com/company/alert-rules",
      "path": "/production",
      "credentials": {
        "username_secret": "githubusername-secret",
        "password_secret": "githubpassword-secret"
      }
    }
  }
}

SSH Auth

dcos security secrets create -f <PATH_TO_PRIVATE_KEY> gitsshkey-secret

For GitHub, you must add the Deployment Key (that is, the public key) to the repository.

Create a custom option file (options.json) like the following.

{
  "prometheus": {
    "alert_rules_repository": {
      "url": "git@github.com:company/alert-rules.git",
      "path": "/production",
      "credentials": {
        "deploy_key_secret": "gitsshkey-secret"
      }
    }
  }
}

NOTE: You will have to use `git@github.com:/.git` instead of https as the scheme of the URL.

Fetching from a branch in a Git repository

By default, the service will fetch from the master branch (that is, refs/heads/master) of the Git repository.

If you want to fetch the alert rules from another branch in a Git repository, you can set the reference_name field:

{
  "prometheus": {
    "alert_rules_repository": {
      "url": "https://github.com/company/alert-rules",
      "path": "/production",
      "reference_name": "refs/heads/my_branch"
    }
  }
}

Install DC/OS Monitoring service

Then, install the service:

dcos package install dcos-monitoring --options=options.json

The Prometheus alert rules defined in the repository will be automatically loaded when the service finishes deploying. You can go to the Prometheus UI to verify.

Triggering a reload of Prometheus alert rules

It is possible to trigger a reload of the Prometheus alert rules after the service is installed.

dcos monitoring plan start reload-prometheus-alert-rules

Alert Rules

Managing alert rules configurations