Configuration Guide - AWS Linux VM
Detailed configuration guide for the Spyderbat AWS Agent installed on an AWS VM
This guide explains how to configure the Spyderbat AWS Agent to collect information from an AWS account and send it to the Spyderbat platform. It provides detailed instructions for locating the configuration file, managing AWS credentials, and configuring all available settings.
1. Managing the configuration
The Spyderbat AWS Agent's configuration file is a YAML file named aws-agent.yaml
. It is used to control the behavior of the agent, such as which AWS services to monitor, where to send data, and how to manage credentials.
Locating the Configuration File
By default, the configuration file is located at:
/opt/spyderbat/etc/aws-agent.yaml
This file can be edited using any text editor with root privileges. For example:
sudo vi /opt/spyderbat/etc/aws-agent.yaml
Applying Changes
After making changes to the configuration file, the AWS Agent service must be restarted to apply the updates. Use the following command to restart the service:
sudo systemctl restart aws_agent.service
Validating Configuration
To ensure the configuration file is valid, check the service status after restarting:
sudo systemctl status aws_agent.service
aws_agent.service - Spyderbat AWS Agent Service
Loaded: loaded (/etc/systemd/system/aws_agent.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-12-11 18:48:48 UTC; 3 weeks 6 days ago
Main PID: 2146512 (aws_agent)
Tasks: 8 (limit: 1112)
Memory: 10.1M
CPU: 4min 7.715s
CGroup: /system.slice/aws_agent.service
├─2146512 /usr/bin/bash /opt/spyderbat/bin/aws_agent
└─2146528 docker run --pull always -v /opt/spyderbat/etc:/etc/aws-config --name aws-agent public.ecr.aws/a6j2k0g1/aws-agent:latest --config /etc/aws->
Jan 08 12:24:30 ip-172-31-86-31.ec2.internal aws_agent[2146528]: 2025-01-08 12:24:30,479:INFO :poller eks/us-west-1 got 2 records
Jan 08 12:24:30 ip-172-31-86-31.ec2.internal aws_agent[2146528]: 2025-01-08 12:24:30,755:INFO :Sending heartbeat and stat update
Jan 08 12:24:31 ip-172-31-86-31.ec2.internal aws_agent[2146528]: 2025-01-08 12:24:31,346:INFO :Session(region_name=None) IAM Poller got 56 roles and their inl>
If there are any errors, they will be displayed in the status output. Ensure the YAML syntax is correct before restarting the service again.
Checking agent logs
The agent logs can be consulted based on the service journal:
sudo journalctl -u aws_agent.service
2. AWS Credentials Management
The Spyderbat AWS Agent requires access to AWS services to collect data. The agent supports multiple methods for obtaining credentials, listed below in the order of precedence:
1. IAM Instance Profile (Recommended)
If the agent is deployed on an EC2 instance with an IAM role assigned, it will automatically use the instance profile credentials. This is the most secure and recommended method. No additional configuration is required for this setup.
2. Environment Variables
You can set the following environment variables to provide credentials explicitly:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
For example, add the variables to the environment:
export AWS_ACCESS_KEY_ID=<your_access_key_id>
export AWS_SECRET_ACCESS_KEY=<your_secret_access_key>
3. From Files
The agent can also read credentials from files. This is typically used when credentials are mounted as secrets in Kubernetes or other containerized environments. Place the credentials in the following files:
/etc/aws-config/secrets/aws_access_key_id
/etc/aws-config/secrets/aws_secret_access_key
Note: This method is not recommended for standalone deployments.
3. Configuration Settings
Below is a detailed explanation of each configuration setting available in the aws-agent.yaml
file.
spyderbat_orc_url
spyderbat_orc_url
Description: The URL of the Spyderbat orchestration API endpoint. This is where the agent sends the collected data.
Example:
spyderbat_orc_url: https://orc.spyderbat.com
Default:
https://orc.spyderbat.com
outfile
outfile
Description: Specifies a file where the agent writes the collected data instead of sending it to the Spyderbat backend. This is primarily for debugging purposes.
Example:
outfile: /tmp/out.json.gz
Default: Not set.
cluster_name
cluster_name
Description: The name of the Kubernetes cluster, used for identification in the Spyderbat UI. This is optional for standalone deployments.
Example:
cluster_name: staging-cluster-us-east-1
Default: Not set.
aws_account_id
aws_account_id
Description: Specifies the AWS account ID the agent monitors. Use
auto
for auto-discovery.Example:
aws_account_id: auto
Default:
auto
role_arn
role_arn
Description: The ARN of the IAM role the agent assumes to gather information. This is useful when explicit AWS credentials are used. It should not be used if the correct role was already assumed through an EC2 IAM Instance Profile.
Example:
role_arn: arn:aws:iam::123456789012:role/SpyderbatRole
Default: Not set.
send_buffer_size
send_buffer_size
Description: The number of records accumulated before sending data to the Spyderbat backend.
Example:
send_buffer_size: 100
Default:
100
send_buffer_records_bytes
send_buffer_records_bytes
Description: The maximum size (in bytes) of accumulated records before sending to the backend.
Example:
send_buffer_records_bytes: 1000000
Default:
1000000
(1 MB)
send_buffer_max_delay
send_buffer_max_delay
Description: The maximum delay (in seconds) before sending accumulated records, even if the buffer is not full.
Example:
send_buffer_max_delay: 30
Default:
30
log_level
log_level
Description: Configures the logging level for the agent.
Options:
DEBUG
,INFO
,WARNING
,ERROR
,CRITICAL
Example:
log_level: INFO
Default:
INFO
pollers
pollers
Description: Configures the AWS services and regions to monitor. Each entry specifies a service, polling interval, and regions.
Example:
pollers: - service: ec2 polling_interval: 30 regions: - us-east-1 - us-west-2 - service: eks polling_interval: 30 regions: - us-east-1 - us-east-2
Default: Monitors all supported services and regions if not set explicitly.
Per service in the pollers section, the following properties can be set:
polling_interval
polling_interval
Description: The interval in seconds at which the agent will poll the service.
Example:
- service: eks polling_interval: 30
Default: 30
regions
regions
Description: The regions that the agent will poll for the service. If not set, the agent will poll all regions.
Example:
- service: eks regions: - us-east-1 - us-east-2
Default: not set (all regions)
Example configuration file
You can find an example illustrated configuration yaml file here
Last updated
Was this helpful?