Notification Configuration Management using Spyctl

Published: December 7, 2023

To Learn more about Notification Configurations see:

Notification Configurations

Prerequisites

If you have never used Spyctl start here to learn how to install it, then follow the Initial Configuration guide.

At least 1 Notification Target is required in order to use Notification Configurations.

Managing Notification Targets

Create

To create a new Notification Configuration you can use the create command:

Note: This will only create a local yaml file for you to edit. It makes no immediate changes to your Spyderbat environment.

spyctl create [OPTIONS] notification-config -n NAME -T TARGET

For example:

spyctl create notification-config -n "Agent Offline" -T OperationsTeam > config.yaml

This creates a completely custom Notification Config. For first-time users we recommend creating Notifications Configs from a template.

This will give you a blank Notification Configuration that looks like this:

config.yaml

apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  # The uid will be generated for you when your run the create command
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX
  name: Agent Offline
  type: object
spec:
  enabled: true
  schemaType: ''
  condition: ''
  target: OperationsTeam
  title: ''
  message: ''
  template: CUSTOM
  additionalFields: {}

To better understand how to configure a custom Notification Config see the full documentation.

Note: for this example to work you would have to have an applied Notification Target named "OperationsTeam"

Creating a Notification Config from a template

Creating a custom Notification Configuration from scratch is a complicated procedure and is recommended only for advanced users. For most use-cases Notification Configs should be created using Notification Configuration Templates.

spyctl create notification-config -n NAME -T TARGET -P TEMPLATE

For example:

spyctl create notification-config -n "Agent Offline" -P nc_tmpl:000000001 -T OperationsTeam > config.yaml

config.yaml

apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX
  name: Agent Offline
  type: object
spec:
  enabled: true
  schemaType: event_opsflag
  subSchema: agent_offline
  condition: ''
  target: OperationsTeam
  title: Spyderbat Agent Detected Offline
  message: '{{ __origin__ }}'
  template: Agent Offline
  additionalFields:
    details:
      Hostname: '{{ hostname }}'
      Time: '{{ __hr_time__ }}'
      Source UID: '{{ muid }}'
      Cluster: '{{ __cluster__ }}'
    linkback_text: View in Spyderbat
    linkback_url: '{{ __linkback__ }}'
    slack_icon: ':red_circle:'

When using a template most of the fields will be filled out for you, but you can still edit them to suit your preference. Refer to the full documentation to understand what each field does.

Edit

When creating new Notification Configs you may want to edit the generated document to customize some of the fields. With spyctl you can use the edit command to ensure you don't accidentally introduce syntax errors.

spyctl edit -f FILENAME

If you have already applied the Notification Config you may edit the resource using the following:

spyctl edit [OPTIONS] notification-config NAME_OR_UID

For example:

spyctl edit -f config.yaml

This will bring up a prompt to select a text editor unless you have already done so previously. Then, using your text editor you may fill in your desired destination or destinations.

If you save without making any changes, nothing happens to the resource or file you're editing. If you save and there were syntax errors, Spyctl will save your draft to a temporary location and re-open it with comments detailing the errors. Finally, if your changes have no syntax errors, Spyctl will update the resource or file you're editing.

Note: If you edit a Notification Config in a local file, but the Config has already been applied. You will need to apply the file again for the updates to take effect.

Apply

In order for a Notification Config to be usable by the Spyderbat Notifications System you must first apply it using the apply command.

spyctl apply -f FILENAME

For example:

spyctl apply -f config.yaml

If the operation is successful, the Spyderbat Notification System will now use your Notification Config to determine when to send you notifications.

Delete

To remove a Notification Config from the Spyderbat Notifications System you can use the delete command.

spyctl delete [OPTIONS] notification-config NAME_OR_UID

For example:

spyctl delete notification-config "Agent Offline"

Disable

Sometimes you might want to temporarily disable a Notification Config. When disabled, the Notification Config will still remain in the Notification System, however no notifications will be generated by it.

Use the edit command to open up a text editor with your Notification Config

spyctl edit notification-config NAME_OR_UID

For example:

spyctl edit notification-config "Agent Offline"

In the text editor, update the enabled field in the Notification Config's spec:

change

...
spec:
  enabled: true
  ...

to

...
spec:
  enabled: false
  ...

As soon as you save, your changes will take effect.

View or Download

You can use the get command to view or download your Notification Configs.

spyctl get [OPTIONS] notification-configs [NAME_OR_UID]

For example:

spyctl get notification-configs

$ spyctl get notification-config
Getting notification-configs
NAME             ID                              TYPE     TARGET            STATUS    AGE
Agent Offline    notif:XXXXXXXXXXXXXXXXXXXXXX    object   OperationsTeam    Enabled   1d

The default output is a tabular summary of your Notification Configs. To download the Notification Config as yaml or json you can use the -o option

spyctl get notification-targets -o yaml [NAME_OR_UID]

$ spyctl get notification-targets -o yaml "Agent Offline"
apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX
  name: Agent Offline
  type: object
spec:
  enabled: true
  schemaType: event_opsflag
  subSchema: agent_offline
  condition: ''
  target: OperationsTeam
  title: Spyderbat Agent Detected Offline
  message: '{{ __origin__ }}'
  template: Agent Offline
  additionalFields:
    details:
      Hostname: '{{ hostname }}'
      Time: '{{ __hr_time__ }}'
      Source UID: '{{ muid }}'
      Cluster: '{{ __cluster__ }}'
    linkback_text: View in Spyderbat
    linkback_url: '{{ __linkback__ }}'
    slack_icon: ':red_circle:'

Using the > character you can save the document to a file.

spyctl get notification-config -o yaml "Agent Offline" > config.yaml

Notification Configuration Templates

Notification Configuration Templates are pre-built documents that serve as a starting point for Notification Configs. Currently, Templates are curated by Spyderbat and exist to help users setup the most commonly desired notifications.

View or Download

You can use the get command to view or download the list of Notification Config Templates.

spyctl get notification-config-templates

$ spyctl get notification-config-templates
Getting notification-config-templates
NAME                            ID                 TYPE          SCHEMA_TYPE    DESCRIPTION
Agent Offline                   nc_tmpl:000000001  agent_health  event_opsflag  Send a notification when Spyderbat detects
                                                                                it is no longer receiving data from an Agent
                                                                                (Nano Agent or Clustermonitor, ephemeral or
                                                                                not).
Agent Back Online               nc_tmpl:000000002  agent_health  event_opsflag  Send a notification when Spyderbat detects
                                                                                that an Agent has come back online and is
                                                                                sending data.
Agent CPU Usage Over Threshold  nc_tmpl:000000003  agent_health  event_metric   Send a notification when an Agent's total
                                                                                CPU usage is over 4% for 2 minutes. (30
                                                                                minute cooldown)
Agent Memory Over Threshold     nc_tmpl:000000004  agent_health  event_metric   Send a notification when an Agent's memory
                                                                                usage is over 3.5GB for 2 minutes. (30
                                                                                minute cooldown)
Agent Bandwidth Over Threshold  nc_tmpl:000000005  agent_health  event_metric   Send a notification when an Agent's
                                                                                bandwidth usage is over 125 KBps for 2
                                                                                minutes. (30 minute cooldown)
# ... The full list is not shown

Usage

Notification Config Templates are used as part of the Notification Config creation process.

spyctl create notification-config --name NAME --template TEMPLATE -target TARGET

For example:

spyctl create notification-config --name "Agent Offline" --template nc_tmpl:000000001 --target OperationsTeam