Ship Stalwart logs to CloudWatch Logs by ryanjjung · Pull Request #209 · thunderbird/mailstrom

ryanjjung · 2026-04-03T19:55:34Z

A few things are happening here. A brief list is below, but I'll leave some commentary inline as well.

Importantly, this cannot be merged until a new release of tb_pulumi is cut. Update: The new code has been released, and we can proceed with this rollout.

Briefly, we:

Build a new log destination for Stalwart logs with two log streams, one for each of our server functions. This helps us separate mail service logs from the logs that show management API activity, which is used programmatically by thunderbird-accounts.
Add a new function concept. This helps us with the api/mail dichotomy. This new parameter in the node definition becomes a tag on the instance, and that gets picked up by the second phase bootstrapping process and templated into the fluent-bit config.
Install fluent-bit as part of the bootstrapping process and install a configuration that extracts only the MESSAGE portion of Thundermail systemd log events and send them along to CloudWatch Logs.

ryanjjung · 2026-04-09T20:28:51Z

pulumi/bootstrap/templates/fluent-bit.service.j2

+
+[Service]
+Type=simple
+Environment="ENV={{ env }}"


{{ this }} is a Jinja variable, replaced in the bootstrapping process. This is how ENV=stage or whatever gets into the service environment.

ryanjjung · 2026-04-09T20:30:07Z

pulumi/bootstrap/templates/fluent-bit.yaml.j2

+    # do not exist, this service must have permission to create them.
+    - name: cloudwatch_logs
+      match: cloudwatch.stalwart.mail
+      log_group_name: /tb/${ENV}/stalwart


${THIS} is not Jinja, but a syntax native to fluent-bit's configuration. This gets subbed out live by fluent-bit with the value of this environment variable.

ryanjjung · 2026-04-09T20:30:30Z

pulumi/bootstrap/templates/fluent-bit.yaml.j2

+pipeline:
+  inputs:
+    - name: systemd
+      tag: cloudwatch.stalwart.{{ function }}


Tagging things with the function lets us catch them later and route them properly.

ryanjjung · 2026-04-09T20:30:57Z

pulumi/bootstrap/bootstrap.py

 # Map of template files to target files
 TEMPLATE_MAP = {
+    'fluent-bit.service.j2': '/usr/lib/systemd/system/fluent-bit.service',
+    'fluent-bit.yaml.j2': '/etc/fluent-bit/fluent-bit.yaml',


Add the new fluent-bit files to the list of things we template on a host.

ryanjjung · 2026-04-09T20:31:15Z

pulumi/bootstrap/bootstrap.py

 # Map of template variable to EC2 tags
 TEMPLATE_VALUE_TAG_MAP = {
+    'env': 'environment',
+    'function': 'postboot.stalwart.function',


These are the new template variables based on instance tags.

ryanjjung · 2026-04-09T20:31:54Z

pulumi/stalwart/__init__.py

        self,
        name: str,
        project: tb_pulumi.ThunderbirdPulumiProject,
+        log_group_arn: str,


This new variable gets passed down to the IAM setup so we can be sure the node profile includes write access to the Stalwart log group.

ryanjjung · 2026-04-09T20:32:27Z

pulumi/stalwart/__init__.py

        archive_files = [
            'bootstrap.py',
+            'templates/fluent-bit.service.j2',
+            'templates/fluent-bit.yaml.j2',


These entries ensure that the new config templates wind up in the first-phase bootstrapping blob.

ryanjjung · 2026-04-09T20:33:13Z

pulumi/config.dev.yaml

        private:
          - destination_cidr_block: 10.202.0.0/22  # observability-dev
-            vpc_peering_connection_id: pcx-0d2027442f0e54ca4
+            vpc_peering_connection_id: pcx-04d7e54008cd9326c


The peering connection changed when I rebuilt dev for testing.

ryanjjung · 2026-04-09T20:35:19Z

pulumi/config.prod.yaml

+  tb:cloudwatch:LogDestination:
+    stalwart:
+      log_group:
+        retention_in_days: 3


Shorter data retention in prod. This is actually the default value for the option, but I like to be explicit.

ryanjjung · 2026-04-09T20:36:15Z

pulumi/requirements.txt

 Jinja2>=3.1,<4.0
 pulumi_cloudflare==6.6.0
-tb_pulumi @ git+https://github.com/thunderbird/pulumi.git@v0.0.16
+tb_pulumi @ git+https://github.com/thunderbird/pulumi.git@v0.0.18


This is a future version with a bunch of little fixes to the core library that we need here, which is why we need the release to be done before merging this.

ryanjjung · 2026-04-09T20:37:31Z

pulumi/stalwart_instance_user_data.sh.j2

 set -x
 set -e

+# Places data get stored


Fun fact: User data can't exceed 16KB. That's why we bzip2 the second phase blob - great compression ratio! This file, fully rendered, is sitting at 7,257 bytes as of this PR, so we're well within range still.

mzeier · 2026-04-09T23:15:30Z

pulumi/bootstrap/templates/fluent-bit.yaml.j2

+  inputs:
+    - name: systemd
+      tag: cloudwatch.stalwart.{{ function }}
+      systemd_filter: _SYSTEMD_UNIT=thundermail.service


Without a persisted cursor (for example DB) or a read_from_tail guard on this systemd input, Fluent Bit will read the existing thundermail.service journal on first boot and can replay entries again after service restarts. That seems likely to backfill or duplicate logs in CloudWatch for these long-lived nodes. Is that intentional?

No, I'll revisit this.

I opted for the db option since it seems more likely to not miss messages.

mzeier · 2026-04-09T23:15:30Z

pulumi/__main__.py

    return stalwart.StalwartCluster(
        f'{project.name_prefix}-stalwart',
        project=project,
+        log_group_arn=logdests['stalwart'].resources['iam_policies']['write'].arn,


This wires the instances to the new LogDestination write policy, but that policy lives in the upcoming tb_pulumi release. From the implementation I reviewed, it looks like logs:CreateLogStream / logs:PutLogEvents may be scoped only to the log-group ARN rather than the log-stream ARNs. If that is still true in the cut release, Fluent Bit will bootstrap successfully but get AccessDenied when it tries to write events. Could we double-check the released policy shape before merging?

No, this was a problem, but that was fixed by this PR, which is slated to go out with that release after it gets approved and merged.

I just got that PR merged and have tested that code directly with success. When I go to prod with it, I'll double-check the policy that goes out there before shipping the fluent-bit configs to the live servers.

ryanjjung · 2026-04-13T15:08:27Z

I cut the new version of tb_pulumi at the end of last week, so we can proceed with this on v0.0.18. I will roll this to stage today, but will wait on review to get it into prod.

ryanjjung · 2026-04-13T15:33:18Z

Just deployed this to stage after setting the tb_pulumi version to v0.0.18 and rebuilding the virtual environment with it. The logs themselves are here, split by server function.

I also double-checked the read/write log policies, and they are correct.

Read:

Write:

ryanjjung · 2026-04-13T16:11:09Z

Let's hang on just a bit. I need to set the journald config to retain fewer logs as well.

ryanjjung · 2026-04-13T16:21:46Z

Ok, this is ready for review again. Vetted in stage.

Add log destination, remove bastion in dev

87e1c13

ryanjjung self-assigned this Apr 3, 2026

ryanjjung added the enhancement New feature or request label Apr 3, 2026

ryanjjung added 8 commits April 6, 2026 14:37

Install fluent-bit to ship logs

39c4d80

Rebuild dev, add env to templates

da7db96

Remove log destination (obsv should have it)

68408cd

Build Stalwart log group; give instances write access

91d7de9

Ship logs directly to CWLogs

0ee694d

Set up stage and prod with logging

04b5a4f

Remove 'untagged' refs

40b5734

Commentary and line

411ac50

ryanjjung commented Apr 9, 2026

View reviewed changes

Reduce log retention in prod

ece6194

ryanjjung commented Apr 9, 2026

View reviewed changes

ryanjjung marked this pull request as ready for review April 9, 2026 20:39

ryanjjung requested review from aatchison and mzeier April 9, 2026 20:43

mzeier reviewed Apr 9, 2026

View reviewed changes

Add journald cursor storage for fluent-bit

9429886

ryanjjung requested a review from mzeier April 10, 2026 15:48

Set log retention on journald as well

9e6b550

Conversation

ryanjjung commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanjjung Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanjjung commented Apr 13, 2026

Uh oh!

ryanjjung commented Apr 13, 2026

Uh oh!

ryanjjung commented Apr 13, 2026

Uh oh!

ryanjjung commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ryanjjung commented Apr 3, 2026 •

edited

Loading

ryanjjung Apr 9, 2026 •

edited

Loading

ryanjjung commented Apr 13, 2026 •

edited

Loading