Book a Demo Start Free Contact Us

Notifications

This section documents the full capabilities of Notification Configurations. It explains the various types of Notification Configs and which fields exist to manipulate their output/behavior.

The Conditions and Triggering Notifications section will briefly explain how the condition field interacts with the Spyderbat data model in order to trigger notifications.

The Dereferencing Values section describes how you can inject values from the json record that triggered the notification into the notification itself.

The Internal Functions section details the various functions you can use to add additional context to your notifications or manipulate existing values into a more desirable format.

Note: The following are advanced concepts. An understanding of them is not required to get started with notifications. Follow the how-to guides to quickly setup commonly-used notifications.

Data Model Primer

Notifications in Spyderbat are driven by the data model. The Spyderbat Nano Agent generates raw telemetry and sends it to the Spyderbat Analytics Engine. The Analytics Engine processes the raw data and builds the behavior web that is viewable in the Console. Additionally, the Analytics Engine analyzes data in the behavior web for security detections, operations issues, policy violations, and more.

The data emitted by the Analytics Engine comes in two flavors: models and events. Notifications are generated by evaluating these two types of records. Models are (potentially) long-lived objects that have a start, middle, and end in their lifecycle. Events represent detections or occurrences that happen at a single point in time.

Take processes as an example. Spyderbat receives process telemetry and builds models to track the state of the processes themselves. What gets emitted from Spyderbat looks like this:

Process Model

{
  "schema": "model_process::1.2.0",
  "id": "proc:_ZO7yNX2S54:ZWn4uw:753728",
  "version": 1701443785,
  "description": "bash [753728, normal] closed from 7dbab6f7-77de-494a-9490-564bc7174611",
  "cgroup": "systemd:/user.slice/user-1000.slice/session-966.scope",
  "time": 1701443785.3360424,
  "create_time": 1701443771.7009835,
  "valid_from": 1701443771.7009835,
  "muid": "mach:_ZO7yNX2S54",
  "pid": 753728,
  "ppid": 753726,
  "ppuid": "proc:_ZO7yNX2S54:ZWn4uw:753726",
  "tpuid": "proc:_ZO7yNX2S54:ZWn4uw:753726",
  "sid": "966",
  "args": [
    "/usr/bin/bash"
  ],
  "cwd": "/home/ubuntu",
  "thread": false,
  "type": "normal",
  "interactive": true,
  "environ": {},
  "duration": 9.437932014465332,
  "name": "bash",
  "title": "/usr/bin/bash",
  "auid": 1000,
  "euid": 0,
  "egid": 0,
  "container": null,
  "auser": "ubuntu",
  "euser": "root",
  "egrp": "root",
  "status": "closed",
  "data_is_complete": true,
  "ancestors": [
    "sudo",
    "bash",
    "sshd",
    "sshd",
    "systemd"
  ],
  "is_causer": false,
  "is_causee": false,
  "prev_time": 1701443781.1389155,
  "expire_at": 1701446399.999999,
  "exit": 0,
  "exe": "/usr/bin/bash",
  "valid_to": 1701443781.1389155,
  "traces": [
    "trace:_ZO7yNX2S54:AAYLdEBuJOo:753573:remote_access"
  ],
  "schemaType": "model_process",
  "schemaMajorVersion": 1,
  "record_type": "model",
  "versionedId": "proc:_ZO7yNX2S54:ZWn4uw:753728:v1701443785"
}

The model above is for an interactive bash shell process running on a machine with the Spyderbat Nano Agent installed. It contains all the information required to add it into the behavior web. It also happens that this process is running with an effective user (euser) "root". That is a privileged account and we have a security detection when we see an interactive shell running as root.

Red Flag Event

{
  "id": "event_alert:_ZO7yNX2S54:ZWn4uw:753728",
  "schema": "event_redflag:root_shell:1.1.0",
  "description": "ubuntu as root ran unusual interactive shell '/usr/bin/bash'",
  "ref": "proc:_ZO7yNX2S54:ZWn4uw:753728",
  "short_name": "root_shell",
  "class": [
    "redflag",
    "proc",
    "root_shell",
    "critical_severity"
  ],
  "flag_class": "redflag/proc/root_shell/critical_severity",
  "severity": "critical",
  "time": 1701443781.1389155,
  "routing": "customer",
  "version": 2,
  "muid": "mach:_ZO7yNX2S54",
  "name": "bash",
  "auid": 1000,
  "args": [
    "/usr/bin/bash"
  ],
  "auser": "ubuntu",
  "euser": "root",
  "ancestors": [
    "sudo",
    "bash",
    "sshd",
    "sshd",
    "systemd"
  ],
  "mitre_mapping": [
    {
      "sub-technique": "T1059.004",
      "sub-technique_name": "Unix Shell",
      "url": "https://attack.mitre.org/techniques/T1059/004",
      "created": "2020-03-09T14:15:05.330Z",
      "modified": "2021-07-26T22:34:43.261Z",
      "stix": "attack-pattern--a9d4b653-6915-42af-98b2-5758c4ceee56",
      "technique": "T1059",
      "technique_name": "Command and Scripting Interpreter",
      "tactic": "TA0002",
      "tactic_name": "Execution",
      "platform": "Linux"
    }
  ],
  "impact": "A shell owned by root has a dangerous level of permissions.",
  "ppuid": "proc:_ZO7yNX2S54:ZWn4uw:753726",
  "false_positive": false,
  "traces": [
    "trace:_ZO7yNX2S54:AAYLdEBuJOo:753573:remote_access"
  ],
  "traces_suppressed": false,
  "schemaType": "event_redflag",
}

The Red Flag above is a security detection on the root bash process above. Events generally have a ref field that points to the id of the model they're related to. This fact is important for notifications as we will see when we discuss the spec fields.

How Notification Configs interact with the data model

There are three fields in the Notification Config spec that determine if a record is a match, schemaType, subSchema, and condition. These fields are explained in more detail below. All records emitted by Analytics have a schema field which is broken into 3 parts:

SCHEMA_TYPE:[SUB_SCHEMA]:VERSION

schemaType, and subSchema in the Notification Config are directly related to the sections of the schema in the records above and must match exactly in order for a notification to fire. condition uses our search syntax to evaluate any of the indexed fields in the records above. If all three of these fields are a match, then a notification will fire assuming its not in cooldown or waiting for a certain duration.

Types

1. Object

Object Notification Configurations are the basic form of Notification Config and work for nearly any type of record emitted by the Spyderbat Analytics Engine.

2. Metrics

Metrics Notification Configurations are specific to event_metric schema types. They work in nearly the same way as Object Notification configs, except for the fact that they have access to a forDuration field. This is because metrics are generated at regular intervals, while that level of predictability is not guaranteed for other objects.

Spec Fields

enabled

  • Type: Boolean

  • Required: Yes

  • Description: Specifies whether the Notification Configuration is active or inactive.

condition

  • Type: String

  • Required: Yes

  • Description: Defines the conditions under which a notification is triggered. This field accepts a string representing the condition. A condition with an empty string will always evaluate as a match.

  • Examples:

    • euser = "root" - For objects with an indexed euser field set to "root"

    • score > 50 AND trigger_short_name ~= "*remote_access*" - For model_spydertrace objects with a score over 50 and a trigger_short_name matching the pattern "*remote_access*" with * being wildcards.

    • "" - If a condition is set as an empty string it will always evaluate to True

Note: if conditions are left as an empty string, it will automatically evaluate to true.

title

  • Type: String

  • Required: Yes

  • Description: Provides a title for the notification, offering a quick summary of the triggered event. This is equivalent to the subject of an email. *Supports Dereferencing Values*

message

  • Type: String

  • Required: No

  • Description: Allows users to customize the content of the notification message. This field is optional. It is the equivalent of the body of an email. *Supports Dereferencing Values*

schemaType

  • Type: String

  • Required: Yes

  • Description: Specifies the type of record evaluated by the Notification Configuration.

  • Examples:

    • event_opsflag - Created when an operations-related event of interest occurs

    • event_redflag- Created when a security-related event of interest occurs

    • event_metric - Created regularly for tracking metrics of things like Agents and Machines

    • model_process - Track the state of individual processes running on a Machine

    • model_connection - Track the state of individual connections to/from/within a Machine

    • model_spydertrace - Track the state of a series of objects related to 1 or more security events.

Note: this is not a complete list. There are many schemaTypes emitted by the Analytics Engine, and more will exist as new features are released.

subSchema

  • Type: String

  • Required: No

  • Description: Defines additional sub-schema for more detailed data structuring within the notification.

  • Examples:

    • for the schema event_opsflag:agent_offline:1.0.0 the subSchema is agent_offline

    • for the schema model_process::1.2.0 there is no subSchema

    • for the schema event_redflag:root_shell:1.1.0 the subSchema is root_shell

target

  • Type: String or List of Strings

  • Required: Yes

  • Description: Determines where the notification should be sent. It can be a single Notification Target or a list of Targets.

template

  • Type: String

  • Required: No

  • Description: The template used to build the Notification Configuration. If omitted this means the Notification is custom.

forDuration

  • Type: Integer

  • Required: No (only available for Metrics Notification Configurations)

  • Description: Sets the duration (in seconds) for which the condition must be true to trigger a notification. This field is optional and applicable only to Metrics Notification Configurations.

cooldown

  • Type: Integer

  • Required: No

  • Description: Specifies the cooldown period (in seconds) between consecutive notifications for the same condition and object. This field is optional.

  • Examples:

    • For event Schema Types, the cooldown is related to the object in the ref field. Take an event_metric:agent:1.0.0 record. It will reference a Nano Agent model. For a Notification Config monitoring for high CPU usage and a cooldown of 30 minutes (1,800 seconds) a Notification will be triggered for every Agent (not every metric event) that goes over the CPU threshold, but the same agent will not trigger a second notification until 30 minutes has passed.

    • For model Schema Types, the cooldown is related to the object's id field. Take a model_agent::1.0.0 record. For a Notification Config monitoring agents in a critical state and a cooldown of 6 hours (21,600 seconds) a Notification will be triggered for every Agent that goes into a critical state, but the same agent will not trigger a second notification until 6 hours has passed.

additionalFields

  • Type: Dictionary

  • Required: No

  • Description: Allows users to include additional optional fields that add context and formatted details to notifications. This field is optional. Refer to the Additional Fields section for more information.

Additional Fields

details

  • Type: Dictionary with String Keys and String Values

  • Required: No

  • Description: Allows users to include detailed information using a dictionary structure. This field is optional. *Supports Dereferencing Values*

  • Examples:

"details": {
   "Severity": {{ severity }},
   "Category": "Error",
   "Description": "Unexpected behavior"
}

linkback_url

  • Type: String (Must be a Valid URL)

  • Required: No

  • Description: Specifies a URL to provide a link back to relevant information or actions. This field is optional. *Supports Dereferencing Values*

  • Examples:

"linkback_url": "https://app.spyderbat.com/app/org/ORG_UID/dashboard"
"linkback_url": "{{ __linkback__ }}"

linkback_text

  • Type: String

  • Required: No

  • Description: Defines the text to display for the link back. This field is optional. *Supports Dereferencing Values*

  • Examples:

"linkback_text": "View in Spyderbat"

slack_icon

  • Type: String

  • Required: No

  • Description: Allows users to set a custom icon for Slack notifications. This field is optional.

  • Examples:

"slack_icon": ":large_green_circle:"
"slack_icon": ":rec_circle:"

Conditions and Triggering Notifications

Dereferencing Values

In Spyderbat's Notification Configurations, you can dynamically include specific values from the JSON objects you are monitoring in the message, title, and additionalFields fields using dereferencing syntax. The syntax for dereferencing is as follows:

  • For direct field access: {{ field_name }}

  • For subfield access within a dictionary: {{ parent_field.sub_field }}

Let's consider an example JSON object:

{
    "id": "event_alert:wPomESruZyzq0LOCbW74:agent_offline",
    "schema": "event_opsflag:agent_offline:1.1.0",
    "description": "Nano Agent timeout detected. Agent was last seen at 1700175960.",
    "ref": "agent:b5elhzkcLeJi3eUapHz7",
    "short_name": "agent_status_offline",
    "class": [
        "opsflag",
        "agent",
        "agent_status",
        "info",
        "offline"
    ],
    "flag_class": "opsflag/agent/agent_status/info/offline",
    "hostname": "build_server",
    "cluster_name": "dev",
    "severity": "info",
    "time": 1700175960.3145692,
    "routing": "customer",
    "version": 1,
    "muid": "mach:YkKvF-XAQis",
    "bat_name": "snapshotUID",
    "ephemeral": false,
    "agent_type": "nano_agent"
}

Examples of Dereferencing Values from the Object Above:

In the message field:

"message": "Alert: {{ description }}. Last seen at {{ time }}."

This would result in a message like: "Alert: Nano Agent timeout detected. Last seen at 1700175960.3145692."

In Title Field:

"title": "Agent for {{ hostname }} in {{ cluster_name }} cluster offline"

This would result in a title like: "Agent for build_server offline in dev cluster offline"

In additionalFields:

additionalFields:
  details: 
    Agent UID: "{{ ref }}"
    Severity: "{{ severity }}"

This would result in additional fields like:

  • Agent UID: agent:b5elhzkcLeJi3eUapHz7

  • Severity: info

These examples demonstrate how you can leverage dereferencing to dynamically include specific values from the JSON object in your notifications. Feel free to adjust the examples based on your specific use cases and requirements.

Internal Functions

In Spyderbat's Notification Configurations, you can enhance the output of your notifications by using internal functions in the title, message, and additionalFields fields. The syntax for using functions is as follows:

  • {{ __FUNCTION_NAME__ [| ARG1, ARG2, ..., ARGN] }}

Arguments are optional, depending on the function used. The return value of the function will replace the {{ __FUNCTION_NAME__ }} placeholder, or an error message will be displayed if something goes wrong.

Example JSON object used in function examples:

This metrics record is used to monitor the resource utilization of the Spyderbat Nano Agent.

{
  "schema": "event_metric:agent:1.0.0",
  "id": "event_metrics:07Ax6uRpB606065sYozQ:ZWXXXX",
  "ref": "agent:07Ax6uRpB606065sXXXX",
  "version": 1,
  "muid": "mach:5sZN4f2mXXX",
  "time": 1701460978.1299076,
  "cpu_cores": 2,
  "total_mem_B": 8173600768,
  "hostname": "example_machine",
  "cluster_name": null,
  "bandwidth_1min_Bps": 1616,
  "cpu_1min_P": {
    "agent": 0.0417,
    "authUID": 0.0002,
    "bashbatUID": 0.0002,
    "grimreaperUID": 0.0034,
    "procmonUID": 0.0003,
    "scentlessUID": 0.0355,
    "snapshotUID": 0
  },
  "mem_1min_B": {
    "agent": 355326000,
    "authUID": 29440000,
    "bashbatUID": 37820000,
    "grimreaperUID": 42692000,
    "procmonUID": 38005000,
    "scentlessUID": 71352000,
    "snapshotUID": 45604000
  },
  "mem_1min_P": {
    "agent": 0.04347239485822658,
    "authUID": 0.003601839732038158,
    "bashbatUID": 0.004627091666633258,
    "grimreaperUID": 0.005223156991853704,
    "procmonUID": 0.004649725510058091,
    "scentlessUID": 0.008729567546208785,
    "snapshotUID": 0.005579425921870522
  },
  "missed_messages": {
    "scentless": {
      "pcap_dropped": 0,
      "data_drop_delayq": 0,
      "data_drop_dns": 0,
      "missed_events": 0
    }
  }
}

Functions:

{{ __cluster__ }}

  • Arguments: This function takes 0 arguments

  • Description: Returns the name of the cluster the object is associated with or "No Cluster."

Example:

additionalFields:
  details: 
    Cluster: "{{ __cluster__ }}"

This would result in a list displayed in the notification:

  • Cluster: No Cluster

This is because in the metrics object above, the cluster_name field is null.

{{ __hr_time__ }}

  • Arguments: This function takes 0 arguments

  • Description: Returns a human-readable version of the time field found in the object.

Example:

additionalFields:
  details: 
    Time: "{{ __hr_time__ }}"

This would result in a list displayed in the notification:

  • Time: 2023-12-01 20:02:58UTC

This converts the epoch time 1701460978.1299076 in the time field in the record above to something human readable.

{{ __linkback__ }}

  • Arguments: This function takes 0 arguments

  • Description: Returns a relevant URL linking back to the Spyderbat Console for the object being evaluated.

Example:

additionalFields:
  linkback_url: "{{ __linkback__ }}"
  linkback_text: "View in Spyderbat"

This would result in a linkback URL being generated, pointing to the Agent Health page for the agent referenced above "ref": "agent:07Ax6uRpB606065sXXXX". It would display as a link "View in Spyderbat" at the bottom of your notification.

{{ __origin__ }}

  • Arguments: This function takes 0 arguments

  • Description: Returns a string explaining why the notification was generated.

Example:

message: "Notification Origin:\n{{ __origin__ }}"

This would result in a message like:

Notification Origin: This notification was generated because an event_metric record matched the condition specified in notification config "Agent CPU Usage - notif:6voXLIYfRPmTky-XVAaXXX".

{{ __percent__ | number }}

  • Arguments: This function takes exactly 1 argument

    • number:

      • type: String or Number

      • description: If a String is supplied, the string must be a field in the object with a numerical value.

  • Description: Multiplies an input number by 100, caps the precision at 2 decimal places, and appends a percent (%) symbol.

Example:

additionalFields:
  details:
    CPU Usage: "{{ __percent__ | cpu_1min_P.agent }}"

This would result in a list displayed in the notification:

  • CPU Usage: 4.17%

Last updated

© SPYDERBAT, Inc., All Rights Reserved