Spyderbat Event Forwarder

Export Spyderbat findings (runtime security flags, Spydertraces) for ingestion into third-party SIEM tools like Splunk, Panther, or Sumo Logic.

Spyderbat Event Forwarder Overview

The event forwarder is a custom utility that consumes Spyderbat events from the API and emits files containing one of the two types of data, security flags or spydertraces. The event forwarder can be integrated with the third-party systems, such as SIEMs and other alert aggregation points, to forward those files via syslog to be consumed outside of Spyderbat platform to further enhance the security content and improve overall observability and security awareness.

For guides detailing the Spyderbat Event Forwarder installation process see this section.

How Does the Spyderbat Event Forwarder Work?

Let’s take a closer look at how the event forwarder collects and transports the data.

Spyderbat analytics system has a specific data store just for redflag events and spydertraces. When a redflag is created on any machine or a spydertrace is kicked off, they get sent to this store. After an agent sees an event, which would create a redflag or trigger a spydertrace, it is stored in the store.

The Spyderbat event forwarder polls the store using Spyderbat API to access these stored events, using a time window of start time and end time in epoch format. It also queries the list of known hosts to augment the events with the details of the known machines.

When the event forwarder is started for the first time, it will read back one hour in time to get the last hour of events, but when it is restarted it will read its logs to figure out the last event it emitted and will use that as its start time. It will read all files which match the file name spyderbat_events*.log in the directory specified by the log_path configuration setting, reading in all the events and keeping the latest time, to use as its starting time.

Once the event forwarder has started, it will either start with reading the last hour of data or start from the last event discovered in the logs (the files named spyderbat_event*.logs). It will set the end time to the current time, so that the query encapsulates a range to ensure that no events are missed.

Once the data comes back from the query, each event is examined to make sure it isn't duplicated and it is augmented to include the details of the associated machines, and the latest event time seen is captured to use as the next start time for the next query.

The event forwarder will then sleep for 30 seconds and then repeats the loop using the latest event time as the start time, but because there is a check to always expand the time window to encapsulate a full 5 minute window, it will always look 4 minutes and 30 seconds further into the past beyond the last event, to ensure that no records have been missed, and will use its LRU (Last Recently Used value) to de-duplicate the events it is capturing.

In the case when there are no events, the start time for the query will not be moved forward until either more events appear, or it reaches its maximum of the 4-hour boundary. This ensures that in the case of an outage on the backend these events will not be missed until they exceed 4 hours of age.

Filtering Data Using Hard-Coded Expressions

It is possible to configure the event forwarder to only log the events that meet certain criteria. This can be achieved through optional filtering using an expression syntax. The expression must evaluate to a bool; If it is true, the event will be logged.

The expression syntax is documented here: https://expr.medv.io/docs/Language-Definition

If the expression fails to compile, the event forwarder will exit with an error at startup. If the expression fails to evaluate, the event will be logged and the forwarder will continue.

The most common reason for an expression to fail to evaluate is that the event does not contain the field(s) referenced in the expression. To avoid this problem, check that the fields you are referencing are not nil, or use the short-circuit "??" operator. Schema is guaranteed to be present.

Here is an example:

expr: |
      (
       schema startsWith "model_spydertrace:"
       and
       (score ?? 0) > 1000
      )
      or
      (
       not
       (
        schema startsWith "model_spydertrace:"
        or
        schema startsWith "event_redflag:bogons:"
        or
        (severity ?? "") in ["info", "low", "medium"]
       )
      )

In this example, the expression will log all events with a schema starting with "model_spydertrace:" and a score greater than 1000. It will log everything else except events with a schema starting with "event_redflag:bogons:" or a severity of "low" or "medium".

Event Forwarder Validation

To ensure that the event forwarder has been deployed and configured correctly and is working as expected, the following conditions must be met and observed:

1. Event forwarder should emit logs to standard out that looks like this: "5 new records, most recent 23s ago". Here is the code reference in the forwarder:\

   log.Printf("%d new records, most recent %v ago", newRecords, et.Sub(lastTime.Time()).Round(time.Second))\

2. Event forwarder should be writing to the specified output in the config file, either to the specified syslog end point or the specified log_path directory with the filename spyderbat_events*.log. You can run 'tail -f spyderbat_events*.log' in that directory to watch the logs being written in real time, you should see events appear within a 10-minute window.