How to Set Up Notifications Using Spyctl

Configure notifications using Spyctl to receive alerts for significant security or operations events.

Published: November 7, 2023

Prerequisites

Overview

Spyderbat's notification system has 3 main components:

Notification Targets: Named destinations to where notifications can be sent.
Notification Configurations: Documents directing Spyderbat on when and where to send notifications. They link to one or more Notification Targets
Notification Configuration Templates: Pre-built templates containing most of the information required to create a Notification Config. These templates simplify the configuration process.

For the the full documentation of the Spyderbat Notifications System refer to this section.

How to Set Up A Notification Target

The first thing to do when setting up notification with Spyderbat is to identify who or what you want to notify be it an email, Slack channel, webhook, or AWS SNS topic. This guide will cover the first 2 options.

All of the commands to manage Notification Targets using Spyctl can be found here.

How to Set Up an Email Target

Create

Use the create command to create a Notification Target YAML document with emails in the --type option:

spyctl create notification-target -n NAME --type emails > FILENAME

for example:

spyctl create notification-target -n OperationsTeam --type emails > target.yaml

This will create a yaml document called target.yaml with a default template for an Email Notification Target

target.yaml

apiVersion: spyderbat/v1
kind: NotificationTarget
metadata:
  uid: notif_tgt:nVefIghnQ2KN3QOEE7jj3Q
  name: OperationsTeam
spec:
  emails:
  - example@example.com

Edit

Email Notification Targets contain a list of 1 or more emails to send notifications to. You can edit this document using the edit command. When using the edit command Spyctl will open your default text editor and automatically validate your document on save.

spyctl edit -f FILENAME

for example:

spyctl edit -f target.yaml

Add your desired email destination(s) to the Notification Target document.

for example:

metadata:
  uid: notif_tgt:nVefIghnQ2KN3QOEE7jj3Q
  name: OperationsTeam
spec:
  emails:
  - engineer1@example.com
  - engineer2@example.com

Once you have added the emails and saved the document you are now ready to apply the Notification Target.

Apply

Use the apply command to save the Notification Target for later use.

spyctl apply -f FILENAME

for example:

spyctl apply -f target.yaml
Successfully applied Notification Target 'notif_tgt:nVefIghnQ2KN3QOEE7jj3Q'

How to Set Up a Slack Target

Create

Use the create command to create a Notification Target YAML document with emails in the --type option:

spyctl create notification-target -n NAME --type slack > FILENAME

for example:

spyctl create notification-target -n OperationsChannel --type slack > target.yaml

This will create a yaml document called target.yaml with a default template for a Slack Notification Target

target.yaml

apiVersion: spyderbat/v1
kind: NotificationTarget
metadata:
  uid: notif_tgt:3j44n_nHTKi7Fk3TeQYcFA
  name: OperationsChannel
spec:
  slack:
    url: https://hooks.slack.com/services/xxxxxxxxxxx/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxx

Edit

Slack Notification Targets point to a Slack Hook URL. You can edit this document using the edit command to add your own URL. When using the edit command Spyctl will open your default text editor and automatically validate your document on save.

spyctl edit -f FILENAME

for example:

spyctl edit -f target.yaml

Add your desired Slack Hook URL destination to the Notification Target document.

Once you have added the URL and saved the document you are now ready to apply the Notification Target.

Apply

Use the apply command to save the Notification Target for later use.

spyctl apply -f FILENAME

for example:

spyctl apply -f target.yaml
Successfully applied Notification Target 'notif_tgt:3j44n_nHTKi7Fk3TeQYcFA'

How to Test a Notification Target

To test that the notification target is configured correctly, use the following command:

spyctl test-notification -T TARGET_NAME

Once this command is run, you should receive a test notification at your specified target in a short amount of time.

How to Set Up Agent Health Notifications

Before going through this section, make sure you have already created at least 1 Notification Target.

All of the commands to manage Notification Configurations using Spyctl can be found here.

Setting up Agent Health notifications is quick and easy using Notification Configuration Templates. To see which notifications can be set up quickly, use the get command to view Agent Health Notification Config Templates.

spyctl get notification-config-templates --type agent-health

This will show you a table of available templates.

$ spyctl get notification-config-templates --type agent-health
Getting notification-config-templates
NAME                            ID                 TYPE          SCHEMA_TYPE    DESCRIPTION
Agent Offline                   nc_tmpl:000000001  agent_health  event_opsflag  Send a notification when Spyderbat detects
                                                                                it is no longer receiving data from an Agent
                                                                                (Nano Agent or Clustermonitor, ephemeral or
                                                                                not).
Agent Back Online               nc_tmpl:000000002  agent_health  event_opsflag  Send a notification when Spyderbat detects
                                                                                that an Agent has come back online and is
                                                                                sending data.
Agent CPU Usage Over Threshold  nc_tmpl:000000003  agent_health  event_metric   Send a notification when an Agent's total
                                                                                CPU usage is over 4% for 2 minutes. (30
                                                                                minute cooldown)
Agent Memory Over Threshold     nc_tmpl:000000004  agent_health  event_metric   Send a notification when an Agent's memory
                                                                                usage is over 3.5GB for 2 minutes. (30
                                                                                minute cooldown)
Agent Bandwidth Over Threshold  nc_tmpl:000000005  agent_health  event_metric   Send a notification when an Agent's
                                                                                bandwidth usage is over 125 KBps for 2
                                                                                minutes. (30 minute cooldown)
Bat Offline                     nc_tmpl:000000006  agent_health  event_opsflag  Send a notification when a Bat goes offline.
Bat Online                      nc_tmpl:000000007  agent_health  event_opsflag  Send a notification when a Bat comes back
                                                                                online.

For this guide we will show you how to setup a notification when one of your Spyderbat Nano Agent's goes offline. You can follow the same steps for any other the other templates you wish to use.

Create the Notification Config

Using the create command you can quickly create a Notification Configuration yaml document.

spyctl create notification-config -n NAME -T TARGET --template TEMPLATE > FILENAME

for example:

spyctl create notification-config -n "Spyderbat Agent Offline" -T OperationsTeam --template nc_tmpl:000000001 > config.yaml

This will create a Notification Configuration from the "Agent Offline" template and look like this:

config.yaml

apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  # A real uid will be generated when you run the create command
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX  
  name: Spyderbat Agent Offline
  type: object
spec:
  enabled: true
  schemaType: event_opsflag
  subSchema: agent_offline
  condition: ''
  target: OperationsTeam
  title: Spyderbat Agent Detected Offline
  message: '{{ __origin__ }}'
  template: Agent Offline
  additionalFields:
    details:
      Hostname: '{{ hostname }}'
      Time: '{{ __hr_time__ }}'
      Source UID: '{{ muid }}'
      Cluster: '{{ __cluster__ }}'
    linkback_text: View in Spyderbat
    linkback_url: '{{ __linkback__ }}'
    slack_icon: ':red_circle:'

Apply the Notification Configuration

At this point no further changes are required. You can use the apply command to save this Config.

spyctl apply -f FILENAME

for example:

spyctl apply -f config.yaml
Successfully applied Notification Config 'notif:XXXXXXXXXXXXXXXXXXXXXX'

At this point a notification will be sent to the Target specified in the Config whenever Spyderbat is no longer receiving data from on of your Agents.

(optional) Add a Second Target

Notifications may be sent to multiple Notification Targets at once. To add a second Target, first ensure you have already configured a second Target, then edit a local file or existing Notification Config with the edit command.

spyctl edit notification-config NAME_OR_UID

for example:

spyctl edit notification-config notif:XXXXXXXXXXXXXXXXXXXXXX

Then update the target field of the Notification Config. This field accepts either a single string or a list of strings.

apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  # A real uid will be generated when you run the create command
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX  
  name: Spyderbat Agent Offline
  type: object
spec:
  ...
  target:
  - OperationsTeam
  - SecurityTeam
  ...

When you save, a notification will be sent to both Targets when the condition is triggered.

Note: If editing a local file, remember to use the apply command for changes to take effect.

(optional) Limit the Scope to a Cluster

You can limit the scope of a Notification Config, such as by cluster. To limit the scope of a Notification Config to a specific cluster edit a local file or existing Notification Config with the edit command.

spyctl edit notification-config NAME_OR_UID

for example:

spyctl edit notification-config notif:XXXXXXXXXXXXXXXXXXXXXX

Then update the condition field of the Notification Config. This field is a string that tells Spyderbat under what conditions to emit a notification. If it is an empty string that defaults to a condition match, otherwise it will evaluate fields within the evaluated JSON object. If the condition is an empty string "" just add in the cluster condition:

apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  # A real uid will be generated when you run the create command
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX  
  name: Spyderbat Agent Offline
  type: object
spec:
  ...
  condition: cluster_name = "dev_cluster"  # or any other name you desire
  ...

If there is an existing condition string, you can add a clause to the condition using the AND keyword.

condition: ephemeral = false AND cluster_name = "dev_cluster"

You can add additional clusters using the OR keyword and ()

condition: ephemeral = false AND (cluster_name = "dev_cluster" OR cluster_name = "staging_cluster")

You can also use pattern matching to scope to multiple clusters. For example, if you have multiple clusters that start with "dev" you can modify the condition like this:

condition: ephemeral = false AND cluster_name ~= "dev*"

When you save, your changes will take effect.

Note: If editing a local file, remember to use the apply command for changes to take effect.

(optional) Limit the Scope to Non-ephemeral Agents

Some Spyderbat Agents are ephemeral by the nature of the underlying machine they are installed on, such as nodes in a Kubernetes cluster. It is common for nodes to be created or deleted on demand and therefore it may be inconvenient to receive a notification every time one goes offline.

To limit the scope of an Agent Health notification to non-ephemeral (permanent) Nano Agents edit a local file or existing Notification Config with the edit command.

spyctl edit notification-config NAME_OR_UID

for example:

spyctl edit notification-config notif:XXXXXXXXXXXXXXXXXXXXXX

apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  # A real uid will be generated when you run the create command
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX  
  name: Spyderbat Agent Offline
  type: object
spec:
  ...
  condition: ephemeral = false
  ...

If there is an existing condition string, you can add a clause to the condition using the AND keyword.

condition: ephemeral = false AND cluster_name = "dev_cluster"

When you save, your changes will take effect.

Note: If editing a local file, remember to use the apply command for changes to take effect.

How to Set Up Security Notifications

Before going through this section, make sure you have already created at least 1 Notification Target.

All of the commands to manage Notification Configurations using Spyctl can be found here.

Setting up Security notifications is quick and easy using Notification Configuration Templates. To see which notifications can be set up quickly, use the get command to view Security-related Notification Config Templates.

spyctl get notification-config-templates --type security

This will show you a table of available templates.

$ spyctl get notification-config-templates --type security
Getting notification-config-templates
NAME                 ID                TYPE      SCHEMA_TYPE    DESCRIPTION
SSH Login Detection  nc_tmpl:00000008  security  event_redflag  Send a notification when Spyderbat detects
                                                                an interactive SSH login.

For this guide we will show you how to setup a notification for when someone performs an interactive SSH login to one of your machines. You can follow the same steps for any other the other templates you wish to use.

Create the Notification Config

Using the create command you can quickly create a Notification Configuration yaml document.

spyctl create notification-config -n NAME -T TARGET --template TEMPLATE > FILENAME

for example:

spyctl create notification-config -n "SSH Connection Detected" -T SecurityTeam --template nc_tmpl:00000008 > config.yaml

This will create a Notification Configuration from the "Agent Offline" template and look like this:

config.yaml

apiVersion: spyderbat/v1
kind: NotificationConfiguration
metadata:
  # A real uid will be generated when you run the create command
  uid: notif:XXXXXXXXXXXXXXXXXXXXXX  
  name: SSH Login
  type: object
spec:
  enabled: true
  schemaType: event_redflag
  condition: ""
  target: SecurityTeam
  title: SSH Login Detected
  message: '{{ description }}


    {{ __origin__ }}'
  template: SSH Login Detection
  additionalFields:
    details:
      Cluster: '{{ __cluster__ }}'
      From IP: '{{ remote_ip }}'
      Logged In As: '{{ logged_in_as }}'
      Source Name: '{{ __source_name__ }}'
      Time: '{{ __hr_time__ }}'
    linkback_text: View in Spyderbat
    linkback_url: '{{ __linkback__ }}'
    slack_icon: ':triangular_flag_on_post:'
  subSchema: interactive_ssh

Apply the Notification Configuration

At this point no further changes are required. You can use the apply command to save this Config.

spyctl apply -f FILENAME

for example:

spyctl apply -f config.yaml
Successfully applied Notification Config 'notif:XXXXXXXXXXXXXXXXXXXXXX'

At this point a notification will be sent to the Target specified in the Config whenever Spyderbat detects an interactive SSH session on one of your machines.

Detailed Documentation

For in-depth documentation on how Notification Configs work go here.

Last updated 1 month ago