Spyderbat allows users to set up Notifications for the below to stay informed about important events in their Spyderbat Organization.

Here are 3 types of notifiable objects:

1. Saved Queries

What it is: Predefined searches to track specific patterns or behaviors in your data.

Why it’s useful: Automates monitoring by notifying you when new activity matches the query.

Example: Get notified when there’s unusual inbound connection.

2. Custom Flags

What it is: Custom flags enable users to create tailored detection rules to monitor activities or behaviors specific to their environment.

Why it’s useful: Helps focus on what matters to you, like unusual commands or risky actions.

Example: Flag and alert when someone runs a command that requires high privileges.

3. Agent Health Notifications

What it is: Alerts about the health and status of Spyderbat agents.

Why it’s useful: Ensures agents are functioning properly and sending data.

Example: Get notified if an agent goes "Offline" or to "Critical" state.

Note: To Learn How to Configure Notifications for Agent Health using Spyctl Refer here

Quick Start Tutorial

To quickly get started using using Spyderbat Notifications follow our tutorial using spyctl.

How to setup Spyderbat Notifications (Spyctl CLI)

Reports can be customized with a variety of input parameters and exported in multiple formats, including JSON, YAML and PDF. Once generated, reports are stored in the "Generated" section for review, export, and printing.

Creating a Report

To create a report:

1. Navigate to the Reports section of the portal and click on the Create menu item.

2. You will see a list of available report templates that you can select from.

Review the descriptions of the reports, and use the preview button to preview a sample of the report type.

1. Select the desired report type from the list.

2. Each report type has specific input parameters you must provide to customize the report for your environment, and specific UI controls to select them.

Enter or select the required parameters, such as:

• Cluster: The Kubernetes cluster for which the report is generated.

• Start Time and End Time: Defines the reporting period (e.g., last 24 hours or specific time range).

You can give your report a specific name to make it easier to locate it later.

Other report types may have other selectors, such as machine selectors.

1. Click Create to initiate the report generation process.

Reports may take several minutes to generate, depending on the size of the system and the selected time range. The UI will display a popup at the bottom of the page titled 'Creating report,' with a 'View' link that directs you to the 'Generated' section.

Viewing Generated Reports

All created reports will appear immediately in the Generated Reports section. Some reports take a while to process and render, and will not be immediately available to view. When they are ready, the 'Published' column will change from 'Scheduled' to the time and date when the report was published, and a 'View' button will be available to allow the user to view the report.

Select the report you want to access and click on its 'View' button. The report will render in a pop-up, like so:

From here you can

1. Inspect the report: Scroll down to see the full report contents if needed

2. Download Report: Export the report in any of the available formats (e.g., JSON, YAML). By clicking on the 'Download' button in the bottom right corner.

3. Export to PDF and print: Use the print to PDF button to export and/or print the report in PDF format using your browser capabilities and preferred settings.

Spyderbat allows you to write Custom Flags using the Spyderbat Query Language (SpyQL). SpyQL enables you to craft precise queries that define the conditions for your Custom Flags.

SpyQL supports complex queries that allow you to combine multiple conditions, use various logical operators (AND, OR, NOT), and apply patterns with matches pattern (~=) and regular expressions with Regex using ~~= operator, equality operator =, etc.

SpyQL is used for Historical Search in Console and for Custom Flags. The queries are composed of two parts, the schema or object type you are looking for and the query itself. In Historical Search you must also specify a time window, however Custom Flags operate in real-time so that section is not supported. The SpyQL query below is from Historical Search and is querying for any Container where the cluster-name field matches the value integrationc2 using the equality operator (=):

You can use Historical Search in the UI to test your Custom Flag queries.

Custom Flags Key Features:

1. Real-Time Monitoring: Once set up, Custom flags operate in real-time, triggering immediate flags when a record matches the SpyQL query.

2. Flexibility: You can define flags that range from broad conditions e.g., anytime a new StatefulSet is created to highly specific scenarios, e.g., a serviceaccount with cluster-admin role created in a particular namespace by a particular user.

3. Red Flag vs Ops Flag: You can choose between a Custom Red Flag (Security) or a Custom Ops Flag (Devops) based on your detection needs.

   • Redflag: Indicates a security issue or potential malicious activity.

   • Opsflag: Highlights operational or configuration issues that may need attention.

   Custom flags also allow you to add your own description and select severity options such as low, high, critical, or info.

4. Integration with Spydertraces: Custom Red Flags may trigger and/or contribute to the score of a Spydertrace just like the built-in Spyderbat detections do.

5. Custom Flag Operations: The ability to create, delete, edit, disable, and enable custom flags further enhances your control over managing the detection process, and gives the ability to evolve as required.

Getting Started

Currently, Custom Flags are only manageable using the Spyctl CLI. Management via the Console UI is coming soon.

Follow the Spyctl CLI tutorial for setting up Custom Flags here.

Workload Policies are tailored to containers and Linux services, specifying a whitelist of permitted activities. This ensures that only known, safe operations are allowed to execute, providing a first line of defense against unauthorized or malicious behavior.

Read more about Workload Policies here.

Key Components:

• A comprehensive list allowed process and network activity.

• Scope: The selectors detailing the specific containers or services to which the policy applies.

• Response: The mechanism by which the policy take actions.

Ruleset Policies

Ruleset Policies offer a more flexible approach, supporting policy-agnostic rulesets that can be applied across different environments. These rulesets contain both allow and deny rules, providing a granular level of control over the behavior within your systems.

Read more about Ruleset Policies here.

Key Components:

• Allow Rules: Explicitly permit certain actions, overriding any broader deny rules that may be in place.

• Deny Rules: Define actions that are explicitly prohibited, regardless of other allow rules.

• Reusability

Interceptor

The Interceptor feature set allows Guardian to take response actions based on policy violations. When a policy violation occurs, Interceptor Response Actions can trigger actions such as generating alerts, or blocking the offending activity.

More details on response actions can be found here.

Tutorials

Tutorials detailing the creation of the various policy types can be found in the tutorials section of this documentation.

• Guardian Tutorials

Conclusion

The Spyderbat Guardian Feature is a powerful tool for maintaining security and compliance in containerized and Linux service environments. By effectively utilizing Guardian Policies, you can ensure that your systems operate within the defined parameters of expected behavior, safeguarding against potential threats.

For more detailed information and advanced configurations, please refer to the policy reference guide.

The agent also supports consuming configured secrets (registration key) in AWS Secrets Manager - which would require an extra permission to access the configured secret arn.

Permissions are configured using a custom AWS policy attached to the IAM Role that the AWS Agent assumes. How the agent assumes this role depends on the deployment options and is discussed in the more detailed deployment guides.

Deployment Options for the AWS Agent

Spyderbat offers multiple deployment options for the AWS Agent to suit different environments and requirements. Below are the currently available deployment methods:

1. Hosted on an AWS VM: You can deploy the AWS Agent on a virtual machine within your AWS account. This option gives you full control over the agent and its environment.

2. Hosted on a Kubernetes Cluster: The AWS Agent can be deployed as a Kubernetes pod within a cluster. This is suitable for users who want to integrate AWS context alongside their Kubernetes workloads.

For detailed installation instructions for each deployment option, refer to the respective guides:

• AWS VM Deployment Guide

• Kubernetes Deployment Guide

Getting Started with the AWS Agent

To begin using the Spyderbat AWS Agent:

1. Choose a Deployment Method: Decide whether to deploy the agent on an AWS VM, a Kubernetes cluster, or use the hosted option.

2. Deploy the Agent: Follow the instructions in the relevant deployment guide to deploy the AWS Agent.

Once deployed, the agent will start collecting cloud context and feeding it to the Spyderbat Platform, where it can be used for enhanced visibility, detection, and investigation.

Enter Spydertop

Spydertop is an open-source tool developed by Spyderbat that provides a solution for this currently unfulfilled use case. Utilizing Spyderbat’s kernel-level system monitoring and public APIs, it provides the same in-depth information as HTOP, and extends these abilities historically. Spydertop allows analysts to look into system anomalies days or even months after they occur.

How it works

Imagine a Kubernetes node that has the Spydertop Nano Agent installed. The agent collects the data necessary for Spydertop to function; for more details, refer to the Installation Guide or watch this video

on how to get it installed. On this system, there happens to be an application with a bug that causes it to continuously use up more memory. At 2:00 in the morning, the container reaches its memory limit and automatic safeguards restart the application. It begins to function correctly afterward, showing no signs of excessive memory usage.

In the morning, an analyst sees the crash report and decides to investigate. They start Spydertop on their own machine, and it uses Spyderbat’s public API to collect all the resource usage records from that early morning crash, as well as the active processes, connections, and more. Using these records, Spydertop displays the memory usage of the machine: 95% at 1:30 AM. By stepping through time, the analyst sees the memory slowly increase until the crash. Next, they sort the running processes by memory usage, find the buggy application, and can now resolve the issue.

How to use Spydertop

You can try out Spydertop by checking out the public repository or running the docker image. If you don’t have an API key yet, it will guide you through setting one up. After that, it is as simple as picking a machine and what time to investigate (both of which can be passed as command-line options for convenience).

Once the necessary data has been loaded from the API, Spydertop presents a simple CLI interface. Spydertop aims to make the transition easy for users already accustomed to HTOP, so the user interface, buttons, and keyboard shortcuts are designed to be similar.

The first few lines display machine-wide resource usage information, such as CPU core usage and disk reads and writes. Taking up the rest of the screen is the process table, which shows resource usage and details for individual processes. Several other tabs are available in this table to show the active sessions, connections, flags, or listening sockets. At the bottom of the screen is a list of quick shortcuts, including a help menu where you can find more detailed information and a list of key binds. A description of command-line options is also available by passing the –help flag.

Get started using Spydertop for free by installing the python CLI, or try it without an account by running the docker image with a set of example data:

Here is the built-in help:

Workload policies shine when you have a relatively stable set of activity. For example, if your organization uses third party or custom containerized applications, each container will typically run a few processes and make a few network connections. This activity can be whitelisted and the policy should stabilize in a short amount of time.

Workload policy do not perform as well for dynamic, constantly changing activity. A development container is a good example of this. When engineers are constantly logging in and running unique and varied commands it becomes impractical to whitelist activity in this way. There will be constant upkeep.

Using Workload Policies

As soon as the Spyderbat Nano Agent is installed on a machine it automatically gathers information of the workload running on it. Spyderbat compiles Fingerprints for each Linux Service and Container it sees. You can think of a Fingerprint as the observed process and network activity for a single workload. Fingerprints are used to build Workload Policies.

Here is an example of a Fingerprint for a single Container:

By default, a Workload Policy is generated by combining all related Fingerprints into a single document. In the image above you can see that this specific container image has 14 instances deployed in our organization. So if one container has network connections that the others do not, the created policy will contain that extra activity.

Once applied, a policy will constantly monitor for deviant activity within the policy's selector scope. In the policy above, that means any container using the docker.io/library/nginx:latest image will be evaluated against this policy.

When deviant activity is detected, multiple things can occur. First, the policy will generate a Deviation. This is a record that contains all of the information required to update the policy with this new activity. Secondly, the policy will take any configured response actions. By default, the policy will generate a Red Flag which is part of Spyderbat's Scout feature. These red flags will become part of Spydertraces that are viewable on your dashboards. Other actions include killing the deviant process, or in Kubernetes you can kill the entire pod.

The script collects the following data in your environment:

1. Summary metrics about the number of nodes, pods, deployments, replicasets, daemonsets, services and namespaces, which helps us assess the size and load on your cluster.

2. Information about the nodes of the cluster, including their provisioned capacity and any taints applied to the nodes, which helps us understand the headroom available in your cluster to add our agents, and helps us pro-actively recommend configuring tolerations on our agents to ensure visibility on all nodes.

3. Cumulative metrics about what resource requests currently running pods are requesting (CPU, memory), which helps us understand the headroom available in your cluster to add our agents.

4. The name and namespaces of the deployments, daemonsets and services running on your cluster, which helps us assess if any other daemonsets or deployments could interfere with our agents and helps us discover if your cluster has node-auto-scaling configured.

5. PriorityClasses currently present for the cluster which helps us assess whether our agent will have sufficient priority to get scheduled on any new nodes being added to the cluster.

The script does NOT collect any of the following:

• Implementation and status details in the 'spec' and 'status' sections of the pods, deployments or daemonsets.

• Any sensitive data that might be present in these sections of the k8s resources (environment variables, configs)

Script Execution Prerequisites

Spyderbat Pre-Deployment Collection script should be run from a machine you currently use to manage your cluster from.

Below are the requirements for the script to run successfully:

1. python3 https://www.python.org/downloads/\

2. kubectl and a valid kube config file https://kubernetes.io/docs/tasks/tools/

   The script will call on the kubectl command to collect cluster information. The cluster(s) to install Spyderbat on should be one of the contexts configured in the kube config file.

Script Execution Steps

First you will need to download the cluster_collect.py script from this public repository.

After installing the script run it as

./cluster_collect.py -h

OR

python3 cluster_collect.py -h\

For usage info run

usage: cluster_collect.py [-h] [-c CONTEXT] [-o OUTPUT]

Here are available options:

• -h, --help show this help message and exit\

• -c CONTEXT, --context CONTEXT kubectl context to pull from (if none provided, all contexts in the kubectl config will be analyzed)\

• -o OUTPUT, --output OUTPUT output file (default is Spyderbat-clusterinfo.json.gz)

By default, the script will collect information for all clusters configured in your kubeconfig file.

If you want to collect only for one cluster, use the -c CONTEXT flag, with the name of the context (as available in kubectl config get-contexts) to collect for.

For example:

./cluster_collect.py -c qacluster1

By default the output will go into a file called spyderbat-clusterinfo.json.gz. You can use the -o flag to use another filename.

Output Delivery and Review

If the script ran successfully, please send the output file back to Spyderbat. We will review the findings with you to discuss the next steps for your deployment and provide recommendations on how to best configure your deployment parameters to ensure that all Spyderbat Nano Agents come online, initialize fully, and successfully register with the Spyderbat backend.

Here is an example of the output file data:

If you would like to review an example of a full file output, please Contact US.

1) The 1st step is to retrieve the command to install the agent for your organization – click on the “New Source” button in the sources section of the product for your organization

2) Once you click on this button, you should be launched into the agent installation wizard where you will be presented with a link to install the agent, let’s copy the “wget” version of the install command and save that to the notepad.

3) Now go to the AWS EC2 management console.

4) Go to Instances and use the Launch Instances wizard to request one or more instances.

5) Choose the desired AMI for the new instances and click Select.

6) Choose the desired instance type. Then click Configure Instance Details.

7) At the bottom of the “Configure Instance Details” screen, you will see an “Advanced Details” section with an input box for “User data”

8) In the user data field, we will enter a shell script to run the install command we copied to our notepad, similar to the below (for RedHat family distributions):

The 1st line indicates this is a bash shell script, the second line ensures the ‘wget’ and ‘lsof’ utilities are installed, and the 3rd line is the install command you copied from the installation wizard. Note that we have omitted “sudo -E” from the command we copied since the user data script is run as root when the instance boots. For Debian family based distributions, the following can be used:

9) Continue with the steps in the install wizard, or jump to Review and Launch if you are done.

10) When the instance is created in AWS, it should now download and install the agent as part of the boot sequence (for reference, the cloud-init output log file is created at /var/log/cloud-init-output.log on the created instance) – note you should ensure the instance(s) that are created have outbound access on port 443 to https://orc.spyderbat.com.

11) Check the “sources” section of the Spyderbat and you should now see your new instance appear in your list of sources.

You can leverage the user data in a similar fashion when using other mechanisms to create AWS EC2 instances, for example when specifying a launch template for an Auto Scaling group.

Click here for more information about Spyderbat’s Nano Agent

Installation Command

Enabling Shell Completion

To enable shell completion, follow these steps:

After modifying the shell config, you need to start a new shell in order for the changes to be loaded.

By default, a monthly quota of 50 is provided, with each trace summary consuming one. You can contact us to request an increase.

What is Summarize?

The Summarize feature in Spyderbat generates a concise summary of a Spydertrace, highlighting critical security insights.

Behind the scenes, it takes the Spydertrace as input, sends it to OpenAI, and generates a concise, easy-to-understand summary.

How to Use Summarize

There are two ways to generate a summary: Manual and Automatic.

1. Manual Summarization

To manually summarize a Spydertrace, click the Summarize button. The summary generation process may take a few seconds.

Example 1: Search

• Search for the relevant Spydertrace.

• If you find a high-score Spydertrace in a restricted cluster, and want to quickly understand its details, click Summarize to generate a summary instantly.

Example 2: Investigation

• Within the Spyderbat Investigation view, click Summarize on top-right to generate a summary.

• Based on the insights, take immediate action as needed.

2. Automatic Summarization

Automatic summarization enables AI-powered summary generation for every Spydertrace saved search.

• When enabled, the system automatically generates structured summaries for saved Spydertrace investigations.

Example:

If you want a summary for every high-score Spydertrace (e.g., score 100), follow these steps:

• Search for the high-score Spydertrace.

• Add it to a saved search.

• Add description, target as desired.

• In Additional Settings, enable Auto AI Summarization and Save.

Once enabled, every time a high-score Spydertrace occurs, you will receive a notification with an investigation link to review the Spydertrace. With automatic summarization, you don't have to wait for the summary to generate—it is ready instantly.

Note: Only enable Automatic AI Summarization based on your organization's quota.

You can also view summarized traces in AI Management's Recent Logs.

Benefits of Summarize

⏳ Time Efficiency

Reduces manual effort in analyzing complex security traces.

⚡ Quick Incident Response

Enables security teams to respond faster with key insights readily available.

🔍 Improved Security Insights

Highlights critical security concerns such as unauthorized access, suspicious executions, and potential breaches.

📑 Simplified Investigation

Provides a structured view of incidents, aiding forensic analysis and remediation planning.

Conclusion

Spyderbat’s Summarize feature enhances security investigations by providing automated, structured, and insightful summaries of activities. By leveraging this feature, security teams can quickly detect, understand, and mitigate potential

• Admin

• Power User

Roles for users can be assigned or modiefied in the Admin section, Organization management. For more details, see User and role management overview

How to Kill a Process or Pod

1. Initiating an Investigation

The process starts with an investigation, which can be the result of pivoting from a high scoring trace, or a security redflag, drilling down from a dashboard card, or just from the output of the search results to find specific proccess or kubernetes resources of interest.

Once you have identified the process of interest in the Investigation section of the UI, you can take action.

In this example, we identified a netcat server running on port 9000

2. Taking the Kill Action

The kill action can be taken either at the bottom of the graph view, or within the process details view. You can opt to kill the process, or, if the process is running in a container from a Kubernetes cluster, to kill the entire pod.

3. Providing a Reason, and Confirming the Kill Action

After selecting the "Kill Process" action, a confirmation dialog will appear, prompting you to provide a reason for the action. This reason will be recorded in the audit log for accountability and future reference. Input a clear and concise reason for the kill action, for example "Terminating malicious process" or "Shutting down compromised pod."

To prevent accidental terminations, Spyderbat requires confirmation before executing the kill action. After entering your reason, click 'YES, KILL PROCESS' to proceed with the process termination.

The process for killing a pod is exactly the same, just select Kill Pod and follow the confirmation prompt.

5. Action Execution

Once confirmed, Spyderbat will automatically terminate the process or pod within seconds. A popup at the bottom-page will provide confirmation.

6. Review and Audit Logging

After a process is killed, its icon will be updated to reflect it is now defunct, and the process details will contain action audit log information

Every kill action, including the reason for the termination and user details, is recorded in Spyderbat’s comprehensive audit log. You can review this log to track all interventions taken during an investigation.

You can find a full log of all actions taken in your account by navigating to the Reports, Action Log section

In the actions log you will find what type of action was taken, the action status, when it was created, the action result code, who took the action, the reason provided and what process or pod was impacted.

You can filter for specific actions you are looking for by clicking on "Filters", and adjust the columns view by clicking on "Columns".

7. Next Steps

After the kill action, you should continue to identify root-cause of the unexpected behavior you chose to terminate.

It is possible another process is still active that could respawn the same type of process you just killed, so reviewing processes again would be a great idea.

Similarly, if you chose to kill the pod, be mindful a new pod might be automatically created by the cluster, exhibiting the same threat you tried to eliminate. Confirm that the behavior you want to see addressed is not introduced by a higher-up kubernetes resource manager, such as a deployment or statefulset that was compromised.

Please check out this section of our portal to learn more about the Spyderbat Nano Agent and the installation details.

Install Event Forwarder via Traditional Installer

Before attempting the install, please, make sure you have downloaded the latest version of the event forwarder (latest release).

• Unpack the tarball:

• Run the installer:

You should see output like this:

• Edit the config file:

• Start the service:

• Check the service:

Use ^C to interrupt the log. If you see errors, check the configuration, restart the service, and check again.

• Enable the service to run at boot time:

• If desired, integrate with the Splunk universal forwarder:

Next Steps

To learn more about the event forwarder and how you can use it to integrate Spyderbat with your other solutions, see this page.

Initial Configuration

In this section you will learn how to configure Spyctl to enable data retrieval from across your entire organization. To do so, you must first create an APISecret and then use that APISecret to set a Context. An APISecret encapsulates your Spyderbat API credentials; the Context specifies where Spyctl should look for data when interacting with the Spyderbat API (e.g., organization, cluster, machine, service, or container image).

Create an APISecret

An APISecret encapsulates your Spyderbat API credentials. You must create at least one APISecret in order for Spyctl to access your data via theSpyderbat API.

To create an APISecret, use an API key generated from the Spyderbat Console.

Copy a generated API key and region-specific API URL into the following command:

For example:

Set a Context

Contexts will let Spyctl know where to look for data. The broadest possible Context is organization-wide. This means that when you run Spyctl commands, the Spyderbat API will return results relevant to your entire organization.

For example:

You can view your configuration by issuing the following command:

You should see something like this:

At this point you should now be able to run spyctl commands that utilize the Spyderbat API.

Let’s take a closer look at how the event forwarder collects and transports the data.

Spyderbat analytics system has a specific data store just for redflag events and spydertraces. When a redflag is created on any machine or a spydertrace is kicked off, they get sent to this store. After an agent sees an event, which would create a redflag or trigger a spydertrace, it is stored in the store.

The Spyderbat event forwarder polls the store using Spyderbat API to access these stored events, using a time window of start time and end time in epoch format. It also queries the list of known hosts to augment the events with the details of the known machines.

When the event forwarder is started for the first time, it will read back one hour in time to get the last hour of events, but when it is restarted it will read its logs to figure out the last event it emitted and will use that as its start time. It will read all files which match the file name spyderbat_events*.log in the directory specified by the log_path configuration setting, reading in all the events and keeping the latest time, to use as its starting time.

Once the event forwarder has started, it will either start with reading the last hour of data or start from the last event discovered in the logs (the files named spyderbat_event*.logs). It will set the end time to the current time, so that the query encapsulates a range to ensure that no events are missed.

Once the data comes back from the query, each event is examined to make sure it isn't duplicated and it is augmented to include the details of the associated machines, and the latest event time seen is captured to use as the next start time for the next query.

The event forwarder will then sleep for 30 seconds and then repeats the loop using the latest event time as the start time, but because there is a check to always expand the time window to encapsulate a full 5 minute window, it will always look 4 minutes and 30 seconds further into the past beyond the last event, to ensure that no records have been missed, and will use its LRU (Last Recently Used value) to de-duplicate the events it is capturing.

In the case when there are no events, the start time for the query will not be moved forward until either more events appear, or it reaches its maximum of the 4-hour boundary. This ensures that in the case of an outage on the backend these events will not be missed until they exceed 4 hours of age.

Filtering Data Using Hard-Coded Expressions

It is possible to configure the event forwarder to only log the events that meet certain criteria. This can be achieved through optional filtering using an expression syntax. The expression must evaluate to a bool; If it is true, the event will be logged.

The expression syntax is documented here: https://expr.medv.io/docs/Language-Definition

If the expression fails to compile, the event forwarder will exit with an error at startup. If the expression fails to evaluate, the event will be logged and the forwarder will continue.

The most common reason for an expression to fail to evaluate is that the event does not contain the field(s) referenced in the expression. To avoid this problem, check that the fields you are referencing are not nil, or use the short-circuit "??" operator. Schema is guaranteed to be present.

Here is an example:

In this example, the expression will log all events with a schema starting with "model_spydertrace:" and a score greater than 1000. It will log everything else except events with a schema starting with "event_redflag:bogons:" or a severity of "low" or "medium".

Event Forwarder Validation

To ensure that the event forwarder has been deployed and configured correctly and is working as expected, the following conditions must be met and observed:

1. Event forwarder should emit logs to standard out that looks like this: "5 new records, most recent 23s ago". Here is the code reference in the forwarder:\

   log.Printf("%d new records, most recent %v ago", newRecords, et.Sub(lastTime.Time()).Round(time.Second))\

2. Event forwarder should be writing to the specified output in the config file, either to the specified syslog end point or the specified log_path directory with the filename spyderbat_events*.log. You can run 'tail -f spyderbat_events*.log' in that directory to watch the logs being written in real time, you should see events appear within a 10-minute window.

Create

To create a new Notification Target you can use the create command:

For example:

This will create a default Notification Target and save it to a file called target.yaml

Edit

When creating new Notification Targets you will need to edit the default document to point to the proper destination. With spyctl you can use the edit command to ensure you don't accidentally introduce syntax errors.

If you have already applied the Notification Target you may edit the resource using the following:

For example:

This will bring up a prompt to select a text editor unless you have already done so previously. Then, using your text editor you may fill in your desired destination or destinations.

If you save without making any changes, nothing happens to the resource or file you're editing. If you save and there were syntax errors, Spyctl will save your draft to a temporary location and re-open it with comments detailing the errors. Finally, if your changes have no syntax errors, Spyctl will update the resource or file you're editing.

Apply

In order for a Notification Target to be usable by the Spyderbat Notifications System you must first apply it using the apply command.

For example:

If the operation is successful, your Notification Target will be ready for use.

Delete

To remove a Notification Target from the Spyderbat Notifications System you can use the delete command.

For example:

View or Download

You can use the get command to view or download your Notification Targets.

For example:

The default output is a tabular summary of your Notification Targets. To download the Notification Target as yaml or json you can use the -o option

Using the > character you can save the document to a file.

Rules are defined as a list in the rules field of a Ruleset's spec.

Each rule contains a target, verb, list of values, and optional selectors (for additional scoping).

• Target: what the rule is referring to within the scope of the policy. Targets are RULE_TYPE::SELECTOR_FIELD.

  • ex. container::image this means that we are allowing or denying containers using images specified in the values field.

• Verb: The currently available verbs for ruleset rules are allow or deny. Any object matching a deny rule will generate a Deviation.

• Values: This is the set of values that are allowed or denied. If the target is container::image then the values should be container images that are either allowed or denied.

• Selectors: Optional selectors that further define the scope of a single rule. For instance you may want a rule that defines allowed activity in a specific namespace within a cluster. Different rule types support different selectors.

Container Rules

Container rules define which containers are allowed or denied.

Examples:

Allow the latest apache image in production and staging

Deny a specific image ID in production

Here is an example permissions policy that can be used when creating the role

Step-by-Step Deployment

Step 1: Launch an AWS VM

Launch an AWS VM within the AWS account you wish to monitor. The instance should be configured with the following settings:

• Amazon Machine Image (AMI): Use an AMI that supports Linux (e.g., Amazon Linux 2, Ubuntu).

• Instance Type: Choose an instance type suitable for your workload (e.g., t3.medium).

• Network Settings: Ensure the instance has access to the internet or appropriate VPC configuration for accessing AWS APIs.

• IAM Role: Attach the IAM Role created earlier with the required permissions.

• Configure storage and other instance details as needed.

Step 2: Connect to the VM and Install Dependencies

1. Install Docker by following the official Docker installation guide.

Step 3: Install the Spyderbat AWS Agent

• Log in to the Spyderbat UI

• Navigate to the Sources menu (top left)

• Click on the Add Source button, and select Install AWS Agent

This will bring you to the following screen:

The agent installation command is obtained from the Spyderbat UI that you can execute on the VM. Click on the tab 'curl' there, and then the command below will be provided that you can paste. If you do not have Curl installed on your system, select the 'wget' tab to copy this command instead. Then use that in the VM to install the agent.

Here's how the curl command will look like

Now execute this script on the AWS VM.

Step 4: Verify Integration

The CLI and UI both provide you with feedback on the process. In the UI, check marks of the install progress will be displayed. Once the Spyderbat AWS Agent is installed, registers with Spyderbat, and is transmitting data, you will see that the agent was installed successfully both in your terminal and in the Spyderbat UI.

Managing the AWS Agent Service

The Spyderbat AWS Agent runs as a systemd service (aws_agent.service) on the VM. You can use the following commands to manage the AWS Agent service:

• Check Service Status:

• Start the Service:

• Stop the Service:

• Restart the Service:

• View Service Logs:

Troubleshooting

• Agent Logs: Check the agent logs using the following command:

• Permission Issues: Ensure the IAM Role attached to the VM has the correct permissions as listed in the prerequisites.

• Network Connectivity: Verify that the VM has access to the internet or the required VPC endpoints to communicate with AWS services.

Next Steps

• Once the AWS Agent is successfully deployed and integrated, you can proceed to use the spyderbat platform to monitor and investigate your assets.

• The AWS Agents behavior can be customized using a configuration file. For more details on advanced configuration of the agent, consult they Spyderbat AWS Agent Configuration Guide

For more details on these configurations, please consult the AWS Agent Configuration Guide for Helm.

Prerequisites

Before deploying the Spyderbat AWS Agent on an AWS EKS cluster, ensure you have the following prerequisites in place:

1. Outbound Network Access: The cluster you’re installing Spyderbat's AWS Agent on must have outbound access on port 443 to https://orc.spyderbat.com.

2. Kubectl and Helm: Install Kubectl and Helm clients, and configure Kubectl for the cluster where you want to install the agent.

3. AWS Account: The cluster the agent is deployed on must reside in the AWS account that you wish to monitor.

4. IAM Role: Create an IAM Role that will be associated with the service account used by the AWS Agent.

   • Role Permissions: The role must have the following permissions attached:

     • EC2: ec2:Describe*

     • EKS: eks:List*, eks:Describe*

   Note that <account-id>, <region>, and <open-id-provider-id> are dependent on your local deployment of the EKS cluster. Take note of the ARN of this role, as it will be an input for the Helm chart deployment.

   You do not need to create the Kubernetes service account associated with the role, as the Helm chart installation will handle that.

Installation with AWS Agent Helm Chart

Step 1 - Copy the Helm install command from the Spyderbat UI

• Log in to the Spyderbat UI

• Navigate to the Sources menu (top left)

• Click on the Add Source button, and select Install AWS Agent

This will bring you to the following screen where you can click on the Helm tab to select installation using the Helm chart.

In the input fields, enter the following:

• Cluster Name: The name of the cluster you are deploying to. This will help the AWS Agent associate itself with the cluster and facilitate recognition in the Cluster Health and Sources UI. This is not required but recommended.

• IAM Role ARN: Enter the ARN of the role you created earlier. This is a required field.

Upon entering the information, the UI will generate a command that you can use to start the installation. Copy the command. It will be similar to the following (your registration key will differ):

Step 2 - Run the Helm Command

In your command-line shell, with Kubectl and Helm installed and configured to use the target cluster as the active context, paste the copied Helm command.

Step 3 - Validate the Installation

Check for any reported errors during the installation, and use the following command to validate that awsagent is installed:

Then use:

You should see a StatefulSet named awsagent-auto and an associated pod named awsagent-auto-0 running if the installation was successful.

To check the logs of the AWS Agent pod, use:

Uninstalling the AWS Agent from Your Cluster

To remove the AWS Agent, use Helm uninstall:

Advanced Configuration

There are various settings that can be customized to address specific needs. These can be achieved by using a custom values.yaml file or by using the --set option during Helm installation.

For more details on these settings, please consult the AWS Agent Configuration Guide for Helm.

If you are monitoring a Kubernetes cluster, you can use a very quick and easy deployment approach via a simple Helm Chart to install the Spyderbat Event Forwarder. It will produce an output to stdout as well as a pvc backed file for easier consumption.

You can access our GitHub public repo to retrieve this Helm Chart here.

You have the following values to override:

To validate if the install was successful, run the following command:

Once run, you should see a similar output to what we have in the example below at the top of the logs followed by any/all events in your organization (possibly filtered if using matching filters) in ndjson format:

Next Steps

To learn more about the event forwarder and how you can use it to integrate Spyderbat with your other solutions, see this page.

In Spyderbat, a schema represents a type of data collected by Spyderbat. For example, the schemas shown in the search UI include Process and Connection, which represent data associated to processes and network connections, respectively.

There are a number of schemas available to search on, each with their own fields that can be queried, which can be found categorized in the schema selector. These schemas are not part of the text of your query, but are selected on the search page. Additionally, a full list of them can be found in the Search Reference.

The Query Builder

After selecting a schema, you can either begin to type an expression, or open the Query Builder. The Query Builder is a useful tool for crafting expressions that contains all the information about the schemas, instead of requiring you to check the reference and documentation.

It contains the full set of fields for each schema, descriptions of the fields, and matches them to comparisons for you, allowing you to easily create effective and correct searches.

The Query Builder will open with the same schema as you had selected in the search page, but you can also select a different schema using the dropdown next to the query preview. Additionally, clicking on the query preview will show a list of recent queries constructed using the Query Builder. Clicking on one will populate the Query Builder and allow you to modify it or send it to the search page.

To modify a query in the Query Builder, use the selector boxes to select a field and what to compare it with. The selectors also contain short descriptions of the fields and operators.

Expressions

After selecting the schema, you’ll need to specify the expression for your search. The expression tells the query engine exactly what criteria to use for filtering data. Every schema has its own set of fields that can be used in the expression, and each field has a type such as String or Number that determines the comparisons available. In addition to these simple types, the List and Map types are also available as composite types, which are discussed below.

Comparisons

The simplest type of expression is a comparison of a field and a value. For example, given that the Process schema has the Executable field, we can use its shortened name exe in an expression to find all processes with the executable "/usr/bin/bash":

This expression uses the = operator, which simply checks for equality with the value. Each field type has a set of available operators in addition to =, allowing for more advanced searches. Additionally, inside an expression, fields must always use their shortened name, and they can only be compared with constant values in the expression itself, not the values of other fields.

A full list of the available comparison operators is available in the Search Reference.

Pattern Matching

String fields have two unique operators: ~= and ~~=, which are Unix glob style pattern matching and regular expression matching, respectively. Regular expressions can be researched elsewhere, but glob pattern matching in this context refers to a string with the characters * and ? used as wildcards. They can both be escaped in a pattern to match their literal characters instead of wildcards using the standard escape character \.

As wildcards, * can match any number of characters, while ? matches exactly one unknown character. For example, we can modify the previous example to check for any bash executable, instead of a specific path:

CIDR Matching

The IP address type also has a CIDR matching operator: <<. CIDR blocks can be researched elsewhere, but the operator is essentialy a way to compare against a range of IP addresses instead of individual ones. For example, we can check for connections from any IP address between 192.168.1.0 and 192.168.1.255:

Logic

Multiple field comparisons can be combined together using the basic logical operations and, or, and not. They can use any capitalization and may be combined with parentheses to specify precedence. For example:

This search uses parentheses to guarantee that the Duration field must always be greater than 60 seconds, regardless of the other conditions.

Field Types

In addition to the available operators, certain types of fields have special rules. For all types, the value used with any comparisons must be of the same type, although Number and Integer are effectively the same. In addition, all String values must be in either single or double quotes, and may escape any ending quotes inside the string using the standard escape character \.

There are two outlier types that fields can have: List and Map. These types both have elements of another type, most commonly String. A List represents an ordered collections of elements, while a Map represents key and element pairs. Interacting with these data types is relatively straightforward and can be combined with other comparisons for more complex queries. To use a List, you can either access elements by index – for example, you would use [0] to access the first element in the list – or search for a value occurring anywhere using the [*] syntax. Elements in a Map can be queried by referencing the key in brackets or with the :keys[*] and :values[*] syntax to search for a key or value anywhere in the map, respectively.

For example, this search finds Process objects with a first argument of “-i”:

Process objects where any argument is “-i”:

Kubernetes Pod objects serving the PostgreSQL database in the "production" namespace:

A full list of fields and their types is available in the Search Reference.

Filtering with related objects

An advanced feature of the Spyderbat search language is related object queries - the ability to reference other objects of different schemas in addition to the main object and query fields on those objects. Every schema has a set of related objects alongside its fields, which point to specific other objects of either the same or a different schema.

After a related object, any field or even related object on the related object's new schema can be chained together using the normal query syntax. This is useful when you want to filter based on information that isn’t contained in the original schema.

The capabilities are best illustrated with an example, such as the machine reference in the Process object, which points to a related object with the Machine schema - in this case, it would be the machine that the process is running on. Using the Cloud Region field in the Machine schema, we can find all bash processes in the "us-east-1" cloud region, even though we do not gather any region data in the Process schema:

In the example above, each process has exactly one associated machine. Some schemas have multiple other objects that can be associated, however, such as the children of a process. In that case, the reference must always be followed by [*], similar to how any element of a List can be used in a comparison. For example, the mentioned children[*] reference in the Process schema can be used to find all bash processes that ran "sudo" in a child process:

Related objects of this type can only be queried with the [*] syntax, and attempting to use an index or a string key will cause a syntax error.

Time Range

After selecting a schema and creating an expression, the last part necessary for a search is to select the time range for the query. When doing so, this will return all data that existed at any point during that time range. For example, a search with the Process schema in a time range of the previous 15 minutes will return all matching processes that were running in the past 15 minutes, but could have been started five minutes or a day ago.

The Spyderbat UI includes a time picker with preset relative times and a custom duration picker. Relative times become constant for a search once it has been run, as opposed to the search continuing to update as time goes by. For a search to continue reporting results, save it to a Dashboard card.

Creation of new AWS IAM Roles

Notification Templates can be referenced while configuring notifications for Notifiable Objects using Spyctl. You can either specify a Notification Target or a Notification Template that map specific targets to templates like below.

Example usage with Spyctl:

Example:

Usage:

The $spyctl notifications configure command allows notifications to be sent using either Custom templates with Targets or directly via Targets (using Default Template).

Types:

Note: Below examples shows YAML Templates, but they can also be generated in JSON format.

Email

Email Notification Templates define the subject and body format for email notifications.

Note: Users must populate subject, body_html, and body_text using placeholders to customize the email content.

Slack

Slack Notification Templates define the message structure for Slack notifications. Notification templates can be generated in YAML or JSON format as desired.

After populating template:

Webhook

Webhook Notification Templates define the payload structure for webhook notifications.

After populating template:

PagerDuty

PagerDuty Notification Templates define the format for alerts sent to PagerDuty.

After populating template:

Use these templates to ensure consistent and structured notifications across different channels.

Placeholder Fields and Dynamic Variables

Some fields in Notification Templates are dynamically calculated and replaced at runtime using placeholders. These placeholders allow real-time data insertion into notification messages.

Understanding Placeholder Fields

Placeholder fields allow dynamic values to be inserted into notification templates. These fields are replaced with actual data when a notification is sent.

They're represent with syntax: __field__

Some Common Spyderbat Internal Placeholder Fields are:

{{ __source__ }} - Source of the event

{{ __cluster__ }} - Cluster where the event occurred

{{ __hr_time__ }} - Human-readable timestamp

{{ __linkback__ }} - Link to view the event in Spyderbat

{{ __time_int__ }} - Timestamp in integer format

{{ __origin__ }} - Origin of the event

{{ __hostname__ }} - Hostname where the event occurred

{{ __percent__ }} - Percentage value related to the event

{{ __pd_severity__ }} - Severity level formatted for PagerDuty

{{ __query_name__ }} - Name of the saved query that triggered the event

Example Usage in Email Body:

Dereferencing Values from the Object:

Static fields or regular placeholders ({{ severity }}, {{ description }}) are fields that are passed directly from the model object. Static text remains unchanged and does not need placeholders.

• {{ severity }} - Severity level of the event

• {{ description }} - Description of the event

By customizing Notification Templates with placeholders, users can ensure notifications provide meaningful and actionable information tailored to their needs.

To learn more about Placeholder fields and Constructing templates Read this

Conclusion

By following this guide, you can create well-structured, dynamic Notification Templates for different destinations. Using placeholders correctly ensures your notifications contain relevant, real-time data.

Manage Notification Templates Using Spyctl

To start creating Templates follow our tutorial using Spyctl : Manage Notification Templates Using Spyctl

Quick Start Tutorial

To quickly get started using using Spyderbat Notifications follow our tutorial using spyctl.

How to setup Spyderbat Notifications (Spyctl CLI)

The create command for custom flags allow you to create a custom detection using Spyderbat Query Language (SpyQL) in Spyctl CLI. Spyctl provides help options (--help) to guide you for every command. To view the help for creating a custom flag, run:

To start, you must select the object you want to generate a flag for. This is done via the --schema option. You can view the list of available search schemas with the spyctl search --describe command.

Next you will want to craft a query for the schema you just selected. Each schema has a number of searchable fields, you can view them with Spyctl using the spyctl search --describe SCHEMA command. For example: spyctl search --describe Process or spyctl search --describe model_process both will retrieve the same results.

Using the above information let's create a simple custom flag for a K8s ReplicaSet having more than 6 replica instances:

Explanation:

• replica-flag - The name of the custom flag.

• --schema "Replicaset" - The schema used for the custom flag. To view available schemas/objects for creating custom flags, run $ spyctl search. The list includes processes, connections, all Kubernetes resource schemas, and more. You can also use model_k8s_replicaset for this option.

  • Note: Custom flags cannot be created for event_deviation, event_opsflag, event_redflag, or model_spydertrace Schemas.

• --query "spec.replicas > 6" - The SpyQL query used for the custom flag. The suggested method is to utilize the search functionality in the UI under the Search Section to identify and test the queries you want to flag. Once identified, you can copy and paste the query as a value for the -q option.

• --type "redflag" - The type of the custom flag. By default, the flag type is set to redflag

• --severity "high"- Specifies the perceived severity level of the flag.

• --description "A ReplicaSet running more than 6 replicas found" - A description of the custom flag.

You can also include other options like --content and --impact for the custom flag. These will show up in the console during an investigation. The YAML configuration generated by the create command will look like the example below. Verify the yaml before applying.

This step only generates the YAML. The next step is to apply this flag.

To apply the custom flag, you have two options:

a. Apply Immediately: Run the same command as above and include the --apply flag to apply the flag immediately.

b. Apply from a File: Save the YAML configuration to a file and then apply it using following command: spyctl apply -f FILENAME

You should get "Successfully applied new custom flag with uid: flag:"* after applying the flag. Once set up, custom flags operate in real-time, triggering immediate flag as the query is met.

2. Get All Custom Flags

To retrieve all custom flags that were created, use the following command:

You'll see a list of custom flags like this:

3. Edit a Custom Flag

You can edit a custom flag if required using the below command, by passing the flag ID or name.

After editing the yaml and saving it, you should see:

Successfully edited custom flag with uid: flag:*

4. Delete a Custom Flag

To remove custom flags that are no longer needed, use the below command:

5. Disable a Custom Flag

To temporarily turn off a custom flag without deleting it, use:

6. Enable a Custom Flag

If you need to re-enable a custom flag that has been disabled, use: