Configuration Guide - AWS Linux VM

Detailed configuration guide for the Spyderbat AWS Agent installed on an AWS VM

This guide explains how to configure the Spyderbat AWS Agent to collect information from an AWS account and send it to the Spyderbat platform. It provides detailed instructions for locating the configuration file, managing AWS credentials, and configuring all available settings.

1. Managing the configuration

The Spyderbat AWS Agent's configuration file is a YAML file named aws-agent.yaml. It is used to control the behavior of the agent, such as which AWS services to monitor, where to send data, and how to manage credentials.

Locating the Configuration File

By default, the configuration file is located at:

/opt/spyderbat/etc/aws-agent.yaml

This file can be edited using any text editor with root privileges. For example:

sudo vi /opt/spyderbat/etc/aws-agent.yaml

Applying Changes

After making changes to the configuration file, the AWS Agent service must be restarted to apply the updates. Use the following command to restart the service:

sudo systemctl restart aws_agent.service

Validating Configuration

To ensure the configuration file is valid, check the service status after restarting:

sudo systemctl status aws_agent.service
aws_agent.service - Spyderbat AWS Agent Service
     Loaded: loaded (/etc/systemd/system/aws_agent.service; enabled; preset: disabled)
     Active: active (running) since Wed 2024-12-11 18:48:48 UTC; 3 weeks 6 days ago
   Main PID: 2146512 (aws_agent)
      Tasks: 8 (limit: 1112)
     Memory: 10.1M
        CPU: 4min 7.715s
     CGroup: /system.slice/aws_agent.service
             ├─2146512 /usr/bin/bash /opt/spyderbat/bin/aws_agent
             └─2146528 docker run --pull always -v /opt/spyderbat/etc:/etc/aws-config --name aws-agent public.ecr.aws/a6j2k0g1/aws-agent:latest --config /etc/aws->

Jan 08 12:24:30 ip-172-31-86-31.ec2.internal aws_agent[2146528]:  2025-01-08 12:24:30,479:INFO    :poller eks/us-west-1 got 2 records
Jan 08 12:24:30 ip-172-31-86-31.ec2.internal aws_agent[2146528]:  2025-01-08 12:24:30,755:INFO    :Sending heartbeat and stat update
Jan 08 12:24:31 ip-172-31-86-31.ec2.internal aws_agent[2146528]:  2025-01-08 12:24:31,346:INFO    :Session(region_name=None) IAM Poller got 56 roles and their inl>

If there are any errors, they will be displayed in the status output. Ensure the YAML syntax is correct before restarting the service again.

Checking agent logs

The agent logs can be consulted based on the service journal:

sudo journalctl -u aws_agent.service

2. AWS Credentials Management

The Spyderbat AWS Agent requires access to AWS services to collect data. The agent supports multiple methods for obtaining credentials, listed below in the order of precedence:

1. IAM Instance Profile (Recommended)

If the agent is deployed on an EC2 instance with an IAM role assigned, it will automatically use the instance profile credentials. This is the most secure and recommended method. No additional configuration is required for this setup.

2. Environment Variables

You can set the following environment variables to provide credentials explicitly:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

For example, add the variables to the environment:

export AWS_ACCESS_KEY_ID=<your_access_key_id>
export AWS_SECRET_ACCESS_KEY=<your_secret_access_key>

3. From Files

The agent can also read credentials from files. This is typically used when credentials are mounted as secrets in Kubernetes or other containerized environments. Place the credentials in the following files:

/etc/aws-config/secrets/aws_access_key_id
/etc/aws-config/secrets/aws_secret_access_key

Note: This method is not recommended for standalone deployments.

3. Configuration Settings

Below is a detailed explanation of each configuration setting available in the aws-agent.yaml file.

`spyderbat_orc_url`

Description: The URL of the Spyderbat orchestration API endpoint. This is where the agent sends the collected data.

Example:

spyderbat_orc_url: https://orc.spyderbat.com

Default: https://orc.spyderbat.com

`outfile`

Description: Specifies a file where the agent writes the collected data instead of sending it to the Spyderbat backend. This is primarily for debugging purposes.
Example:
```
outfile: /tmp/out.json.gz
```
Default: Not set.

`cluster_name`

Description: The name of the Kubernetes cluster, used for identification in the Spyderbat UI. This is optional for standalone deployments.
Example:
```
cluster_name: staging-cluster-us-east-1
```
Default: Not set.

`aws_account_id`

Description: Specifies the AWS account ID the agent monitors. Use auto for auto-discovery.
Example:
```
aws_account_id: auto
```
Default: auto

`role_arn`

Description: The ARN of the IAM role the agent assumes to gather information. This is useful when explicit AWS credentials are used. It should not be used if the correct role was already assumed through an EC2 IAM Instance Profile.

Example:

role_arn: arn:aws:iam::123456789012:role/SpyderbatRole

Default: Not set.

`send_buffer_size`

Description: The number of records accumulated before sending data to the Spyderbat backend.
Example:
```
send_buffer_size: 100
```
Default: 100

`send_buffer_records_bytes`

Description: The maximum size (in bytes) of accumulated records before sending to the backend.
Example:
```
send_buffer_records_bytes: 1000000
```
Default: 1000000 (1 MB)

`send_buffer_max_delay`

Description: The maximum delay (in seconds) before sending accumulated records, even if the buffer is not full.
Example:
```
send_buffer_max_delay: 30
```
Default: 30

`log_level`

Description: Configures the logging level for the agent.
Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
Example:
```
log_level: INFO
```
Default: INFO

`pollers`

Description: Configures the AWS services and regions to monitor. Each entry specifies a service, polling interval, and regions.

Example:

pollers:
  - service: ec2
    polling_interval: 30
    regions:
      - us-east-1
      - us-west-2
  - service: eks
    polling_interval: 30
    regions:
      - us-east-1
      - us-east-2

Default: Monitors all supported services and regions if not set explicitly.

Per service in the pollers section, the following properties can be set:

`polling_interval`

Description: The interval in seconds at which the agent will poll the service.

Example:

  - service: eks
    polling_interval: 30

Default: 30

`regions`

Description: The regions that the agent will poll for the service. If not set, the agent will poll all regions.

Example:

  - service: eks
    regions:
      - us-east-1
      - us-east-2

Default: not set (all regions)

Example configuration file

Last updated 3 months ago

Was this helpful?

Configuration Guide - AWS Linux VM

Detailed configuration guide for the Spyderbat AWS Agent installed on an AWS VM

1. Managing the configuration

Locating the Configuration File

By default, the configuration file is located at:

/opt/spyderbat/etc/aws-agent.yaml

This file can be edited using any text editor with root privileges. For example:

sudo vi /opt/spyderbat/etc/aws-agent.yaml

Applying Changes

After making changes to the configuration file, the AWS Agent service must be restarted to apply the updates. Use the following command to restart the service:

sudo systemctl restart aws_agent.service

Validating Configuration

To ensure the configuration file is valid, check the service status after restarting:

sudo systemctl status aws_agent.service
aws_agent.service - Spyderbat AWS Agent Service
     Loaded: loaded (/etc/systemd/system/aws_agent.service; enabled; preset: disabled)
     Active: active (running) since Wed 2024-12-11 18:48:48 UTC; 3 weeks 6 days ago
   Main PID: 2146512 (aws_agent)
      Tasks: 8 (limit: 1112)
     Memory: 10.1M
        CPU: 4min 7.715s
     CGroup: /system.slice/aws_agent.service
             ├─2146512 /usr/bin/bash /opt/spyderbat/bin/aws_agent
             └─2146528 docker run --pull always -v /opt/spyderbat/etc:/etc/aws-config --name aws-agent public.ecr.aws/a6j2k0g1/aws-agent:latest --config /etc/aws->

Jan 08 12:24:30 ip-172-31-86-31.ec2.internal aws_agent[2146528]:  2025-01-08 12:24:30,479:INFO    :poller eks/us-west-1 got 2 records
Jan 08 12:24:30 ip-172-31-86-31.ec2.internal aws_agent[2146528]:  2025-01-08 12:24:30,755:INFO    :Sending heartbeat and stat update
Jan 08 12:24:31 ip-172-31-86-31.ec2.internal aws_agent[2146528]:  2025-01-08 12:24:31,346:INFO    :Session(region_name=None) IAM Poller got 56 roles and their inl>

If there are any errors, they will be displayed in the status output. Ensure the YAML syntax is correct before restarting the service again.

Checking agent logs

The agent logs can be consulted based on the service journal:

sudo journalctl -u aws_agent.service

2. AWS Credentials Management

The Spyderbat AWS Agent requires access to AWS services to collect data. The agent supports multiple methods for obtaining credentials, listed below in the order of precedence:

1. IAM Instance Profile (Recommended)

2. Environment Variables

You can set the following environment variables to provide credentials explicitly:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

For example, add the variables to the environment:

export AWS_ACCESS_KEY_ID=<your_access_key_id>
export AWS_SECRET_ACCESS_KEY=<your_secret_access_key>

3. From Files

/etc/aws-config/secrets/aws_access_key_id
/etc/aws-config/secrets/aws_secret_access_key

Note: This method is not recommended for standalone deployments.

3. Configuration Settings

Below is a detailed explanation of each configuration setting available in the aws-agent.yaml file.

`spyderbat_orc_url`

Description: The URL of the Spyderbat orchestration API endpoint. This is where the agent sends the collected data.

Example:

spyderbat_orc_url: https://orc.spyderbat.com

Default: https://orc.spyderbat.com

`outfile`

Description: Specifies a file where the agent writes the collected data instead of sending it to the Spyderbat backend. This is primarily for debugging purposes.
Example:
```
outfile: /tmp/out.json.gz
```
Default: Not set.

`cluster_name`

Description: The name of the Kubernetes cluster, used for identification in the Spyderbat UI. This is optional for standalone deployments.
Example:
```
cluster_name: staging-cluster-us-east-1
```
Default: Not set.

`aws_account_id`

Description: Specifies the AWS account ID the agent monitors. Use auto for auto-discovery.
Example:
```
aws_account_id: auto
```
Default: auto

`role_arn`

Description: The ARN of the IAM role the agent assumes to gather information. This is useful when explicit AWS credentials are used. It should not be used if the correct role was already assumed through an EC2 IAM Instance Profile.

Example:

role_arn: arn:aws:iam::123456789012:role/SpyderbatRole

Default: Not set.

`send_buffer_size`

Description: The number of records accumulated before sending data to the Spyderbat backend.
Example:
```
send_buffer_size: 100
```
Default: 100

`send_buffer_records_bytes`

Description: The maximum size (in bytes) of accumulated records before sending to the backend.
Example:
```
send_buffer_records_bytes: 1000000
```
Default: 1000000 (1 MB)

`send_buffer_max_delay`

Description: The maximum delay (in seconds) before sending accumulated records, even if the buffer is not full.
Example:
```
send_buffer_max_delay: 30
```
Default: 30

`log_level`

Description: Configures the logging level for the agent.
Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
Example:
```
log_level: INFO
```
Default: INFO

`pollers`

Description: Configures the AWS services and regions to monitor. Each entry specifies a service, polling interval, and regions.

Example:

pollers:
  - service: ec2
    polling_interval: 30
    regions:
      - us-east-1
      - us-west-2
  - service: eks
    polling_interval: 30
    regions:
      - us-east-1
      - us-east-2

Default: Monitors all supported services and regions if not set explicitly.

Per service in the pollers section, the following properties can be set:

`polling_interval`

Description: The interval in seconds at which the agent will poll the service.

Example:

  - service: eks
    polling_interval: 30

Default: 30

`regions`

Description: The regions that the agent will poll for the service. If not set, the agent will poll all regions.

Example:

  - service: eks
    regions:
      - us-east-1
      - us-east-2

Default: not set (all regions)

Example configuration file

You can find an example illustrated configuration yaml file

Last updated 3 months ago

Was this helpful?