Skip to content

S3 log support#32

Open
mheffner wants to merge 13 commits intomainfrom
s3-log-support
Open

S3 log support#32
mheffner wants to merge 13 commits intomainfrom
s3-log-support

Conversation

@mheffner
Copy link
Contributor

@mheffner mheffner commented Feb 20, 2026

This introduces the framework for supporting S3 event notifications. S3 event notifications will invoke the Lambda function when an AWS service writes new logs to an S3 bucket (the S3 creation event). While this PR mostly lays the framework for future support of S3 log based AWS services, it currently supports CloudTrail logs from S3 (in addition to the existing Cloudtrails CloudWatch support).

A single S3 notification may contain multiple updates from the creation of multiple S3 objects, each which needs to be read, parsed and converted. By default this will process five S3 objects concurrently and will emit logs in batches up to 1k to the logs pipeline. This may be mean that ordering of the S3 objects, and logs, are not maintained. S3 objects listed later in the event notification may be loaded, parsed and exported before earlier S3 objects. In theory this can be fixed by setting FORWARDER_S3_MAX_PARALLEL_OBJECTS=1. However, S3 event notifications may be fired to multiple Lambda methods concurrently, so ordering is not guaranteed on prinicple.

There's some refactoring that I plan to do later that will cleanup sharing between the Cloudwatch and S3 logs support. In addition, the CW logs support should be broken out of the parse module, similar to this new s3logs support.

The PR required some changes to the acker component because we won't know how many acks exist ahead of time. Instead of buffering all acks, we spawn a listener to consume the acks/nacks concurrent to processing the S3 objects.

Fixes: #5

@mheffner mheffner marked this pull request as ready for review February 25, 2026 22:28
@mheffner mheffner requested a review from rjenkins February 25, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support S3 log sources

1 participant