EDOT Cloud Forwarder for AWS

Serverless Observability Preview EDOT CF AWS Preview

EDOT Cloud Forwarder (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Observability on Elastic Cloud Serverless.

EDOT Cloud Forwarder for AWS supports the following log sources:

Read on to learn how to set up EDOT Cloud Forwarder for AWS.

To collect logs using EDOT Cloud Forwarder for AWS, you need:

In addition, you need to know the URL of the managed OTLP endpoint and the API key for authentication.

Before deploying EDOT Cloud Forwarder for AWS, keep these points in mind:

• Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time.
• Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.
• The CloudFormation stack deployment region must match the region of the S3 bucket.

Download the CloudFormation template to deploy the appropriate stack based on your log source:

For specific versions, edit latest in the URL to the required version in the format vX.Y.Z.

EDOT Cloud Forwarder for AWS uses a CloudFormation template to deploy the EDOT Collector as a Lambda function.

These are the required settings you need:

Set the following settings based on the log source:

These are optional settings you can set in the CloudFormation template:

Default values of EdotCloudForwarderMemorySize and EdotCloudForwarderConcurrentExecutions are sufficient for most use cases. Key indications for the need of tuning these parameters are Lambda throttling and Lambda timeouts. Along with these, you might also need to adjust EdotCloudForwarderExporterMaxQueueSize to export higher data volumes.

The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation. Copy and paste these commands after replacing the placeholder values with your actual configuration.

• Use the --template-url flag to reference a template hosted on S3.
• To always use the most recent stable templates, use the latest path. For example, v0/latest.
• To pin a specific version, replace latest with the desired version tag. For example, v0/v{{version.edot-cf-aws}}.

Alternatively, if you have downloaded the template file, use the --template-body file://<path> option with a local template file.

aws cloudformation create-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow"
		

aws cloudformation create-stack \
  --stack-name edot-cloud-forwarder-alb \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elbaccess"
		

To update an existing CloudFormation stack while preserving some parameter values, follow these steps:

You can deploy the stack manually using the AWS Management Console by following these steps:

1. Navigate to CloudFormation in the AWS Console.
2. Select Create Stack and choose With new resources (standard) to start a fresh deployment.
3. Select one of the following options under Specify template:
   • Amazon S3 URL (Recommended): Use the pre-hosted CloudFormation template stored in the public S3 bucket.
   • Upload a template file: Download the template from the S3 bucket and upload it manually.
4. Select Next and configure all required parameters.
5. Select Next again and check Acknowledge IAM capabilities. This is required because the template creates named IAM roles with permissions to access the required resources.
6. Review your configuration and select Submit to deploy the stack.
7. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

The CloudFormation templates create a number of resources to process logs from a specific log source.

This is a list of resources created by the stack when processing S3 logs.

The main Lambda function, LambdaFunction, is the core component for processing S3 logs. S3 event notifications are handled dynamically using CustomNotificationUpdater and NotificationUpdaterLambda.

CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while S3FailureBucketARN prevents data loss by capturing unprocessed logs.

After EDOT Cloud Forwarder for AWS is successfully running and forwarding logs to Elastic Observability, install the Kibana integrations to visualize your data with out-of-the-box dashboards and visualizations.

To set up data visualization in Kibana:

1.Log into your Elastic Cloud deployment and open Kibana. 2. Go to Management → Integrations in the Kibana navigation menu. 3. Search for the appropriate integration based on your log type and install it:

4. Once installed, navigate to Dashboard to view the pre-built dashboards for your AWS log data.

EDOT Cloud Forwarder store Lambda invocation events related to retryable errors at the S3 bucket specified by S3FailureBucketARN parameter. Retryable errors here include,

• Network errors when attempting to forward to OTLPEndpoint
• Invalid or expired ElasticApiKey
• Lambda triggered by events that mismatch EdotCloudForwarderS3LogsType selection

These errors can be replayed manually to back-fill any gaps in your data.

To replay errors invoke the Lambda with manual trigger type replayFailedEvents. Replace <LAMBDA_NAME> with the name from your deployment.

aws lambda invoke \
  --function-name <LAMBDA_NAME> \
  --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \
  --cli-binary-format raw-in-base64-out /dev/null
		

The following settings are available:

When successful, you should get "StatusCode": 200 as the output. Check CloudWatch logs (resource LambdaLogGroup) for detailed logs.

If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command:

aws cloudformation delete-stack \
  --stack-name <stack-name> \
  --region <stack-region>
		

Deleting a stack will remove all AWS resources created by that stack. However, if you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket will not be deleted if it contains objects, because CloudFormation doesn't force-delete non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it.

If you specified an existing bucket through the S3FailureBucketARN parameter instead, that bucket will not be deleted because it is not managed by the stack.

aws cloudformation delete-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --region eu-central-1
		

aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --region <stack-region>
		

aws cloudformation delete-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --region eu-central-1 \
  --deletion-mode FORCE_DELETE_STACK