Skip to content

Add first-class WAL footer version and auto-derived storage URIs #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Alex Soldatchenko (soldatchenko) wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Add first-class WAL footer version and auto-derived storage URIs #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
e522f40
init for no_pg
jeffmccollum Mar 21, 2026
029898e
update pic
jeffmccollum Mar 26, 2026
e798721
move no-pg section
jeffmccollum Mar 26, 2026
9716e62
add post upgrade and more docs
jeffmccollum Mar 27, 2026
09d5b12
extend test
jeffmccollum Mar 27, 2026
a8d1857
add the why for the upgrade
jeffmccollum Mar 27, 2026
b9574c0
Initial commit
soldatchenko Apr 10, 2026
89706c3
Update AGENTS.md and README
soldatchenko Apr 10, 2026
72ad39c
Remove images
soldatchenko Apr 10, 2026
2367472
Remove customer reference from AGENTS
soldatchenko Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,32 @@ Always use the namespace helper:
namespace: {{ include "braintrust.namespace" . }}
```

## Critical Safety Constraints

These constraints exist because of real incidents and confirmed engineering guidance. Do not "simplify" or "clean up" code that implements them.

### Upgrade Sequencing

- **Never set `skipPgForBrainstoreObjects` on Data Plane versions before 2.0.** A known bug on 1.1.32 was hit by a customer and fixed in the 2.0 images. The correct sequence is: 1.1.32 -> WAL v1 -> 2.0 + WAL v3 -> no-PG.
- **Never set `brainstoreWalFooterVersion` in the same deploy as an image version bump.** Old Brainstore nodes still rolling out cannot read the new WAL format. Exception: bumping v1 to v3 can be done in the same deploy as the 2.0 image upgrade because all 2.0 nodes understand v3.
- **`skipPgForBrainstoreObjects` is a one-way operation.** Once enabled for an object type, it cannot be rolled back without downtime. Do not default this to any non-empty value.

### WAL_USE_EFFICIENT_FORMAT Decoupling

`BRAINSTORE_WAL_USE_EFFICIENT_FORMAT` is intentionally derived from EITHER `brainstoreWalFooterVersion` OR `skipPgForBrainstoreObjects` being set. This is not redundant - it enables efficient format as early as possible in the upgrade sequence (when WAL v1 is set) rather than waiting for no-PG. Do not "simplify" this to only check one condition.

### Brainstore ConfigMap Consistency

The three brainstore configmaps (`brainstore-reader-configmap.yaml`, `brainstore-writer-configmap.yaml`, `brainstore-fastreader-configmap.yaml`) must have identical environment variable logic for `BRAINSTORE_RESPONSE_CACHE_URI`, `BRAINSTORE_CODE_BUNDLE_URI`, `BRAINSTORE_ASYNC_SCORING_OBJECTS`, and `BRAINSTORE_LOG_AUTOMATIONS_OBJECTS`. If you modify one, you must update all three.

### Version Numbers

Chart version numbers are semantically meaningful for the upgrade path:
- Minor versions (e.g., 5.2.0) are additive, non-breaking chart features
- Major versions (e.g., 6.0.0) carry Data Plane image tag changes that affect runtime behavior

Do not bump `Chart.yaml` version without explicit coordination - version numbers determine customer upgrade sequencing.

## Review guidelines

When reviewing PRs, verify:
Expand Down
10 changes: 10 additions & 0 deletions braintrust/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,16 @@ You may have to adjust your helm values.yaml overrides if you have adjusted any

Also if you have custom readiness checks, please unset these customizations and use our new default readiness checks. There is a bug in the dataplane where the endpoint we were using for readiness checks, would never recover if it failed.

### Version 6

This version introduces opt-in no-PG mode, allowing Brainstore to store objects directly without PostgreSQL. The new `skipPgForBrainstoreObjects` value defaults to `""` (disabled), so upgrading makes no behavioral change unless you explicitly set it. This solves a longstanding bottleneck - the rate and volume of data ingestion is no longer limited by Postgres, and this means faster, more reliable data ingestion at higher scale.

> **WARNING: no-PG must only be enabled after upgrading to Data Plane 2.0 images.** A known bug on 1.1.32 was fixed in the 2.0 images. Do not enable no-PG on 1.1.32.

> **WARNING: This is a one-way operation.** Once an object type has been migrated off PostgreSQL, it cannot be un-migrated without downtime.

This version also adds first-class `brainstoreWalFooterVersion` support and auto-derived `BRAINSTORE_RESPONSE_CACHE_URI` / `BRAINSTORE_CODE_BUNDLE_URI` from existing `objectStorage` config.

## Example Values Files

Example values files for different cloud providers and configurations are located in the `examples/` folder.
222 changes: 0 additions & 222 deletions braintrust/UPGRADE.md
View file
Open in desktop

This file was deleted.

11 changes: 10 additions & 1 deletion braintrust/templates/api-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,5 +51,14 @@ data:
BRAINSTORE_INSERT_ROW_REFS: "true"
# Logs v2 table. Requires pg_partman extension (default enabled in our TF modules)
INSERT_LOGS2: "true"
ALLOW_INVALID_BASE64: {{ .Values.api.allowInvalidBase64 | default "false" | quote }}
ALLOW_INVALID_BASE64: {{ .Values.api.allowInvalidBase64 | default "false" | quote }}
NODE_MEMORY_PERCENT: {{ .Values.api.nodeMemoryPercent | default "80" | quote }}
{{- if .Values.brainstoreWalFooterVersion }}
BRAINSTORE_WAL_FOOTER_VERSION: {{ .Values.brainstoreWalFooterVersion | quote }}
{{- end }}
{{- if .Values.skipPgForBrainstoreObjects }}
SKIP_PG_FOR_BRAINSTORE_OBJECTS: {{ .Values.skipPgForBrainstoreObjects | quote }}
{{- end }}
{{- if or .Values.brainstoreWalFooterVersion .Values.skipPgForBrainstoreObjects }}
BRAINSTORE_WAL_USE_EFFICIENT_FORMAT: "true"
{{- end }}
20 changes: 20 additions & 0 deletions braintrust/templates/brainstore-fastreader-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,38 @@ data:
{{- if eq .Values.brainstore.locksBackend "objectStorage" }}
BRAINSTORE_LOCKS_URI: "az://{{ .Values.objectStorage.azure.brainstoreContainer }}/brainstore/locks"
{{- end }}
{{- if .Values.objectStorage.azure.responseContainer }}
BRAINSTORE_RESPONSE_CACHE_URI: "az://{{ .Values.objectStorage.azure.responseContainer }}"
Copy link
Copy Markdown
Contributor

@mdeeks Mike Deeks (mdeeks) Apr 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think this is right? There should be a path I think? Can you please double and triple check all of these? If we get them wrong it is essentially impossible to fix later.

{{- end }}
{{- if .Values.objectStorage.azure.codeBundleContainer }}
BRAINSTORE_CODE_BUNDLE_URI: "az://{{ .Values.objectStorage.azure.codeBundleContainer }}"
{{- end }}
{{- else if eq .Values.cloud "aws" }}
BRAINSTORE_INDEX_URI: "s3://{{ .Values.objectStorage.aws.brainstoreBucket }}/brainstore/index"
BRAINSTORE_REALTIME_WAL_URI: "s3://{{ .Values.objectStorage.aws.brainstoreBucket }}/brainstore/wal"
{{- if eq .Values.brainstore.locksBackend "objectStorage" }}
BRAINSTORE_LOCKS_URI: "s3://{{ .Values.objectStorage.aws.brainstoreBucket }}/brainstore/locks"
{{- end }}
{{- if .Values.objectStorage.aws.responseBucket }}
BRAINSTORE_RESPONSE_CACHE_URI: "s3://{{ .Values.objectStorage.aws.responseBucket }}"
{{- end }}
{{- if .Values.objectStorage.aws.codeBundleBucket }}
BRAINSTORE_CODE_BUNDLE_URI: "s3://{{ .Values.objectStorage.aws.codeBundleBucket }}"
{{- end }}
{{- else if eq .Values.cloud "google" }}
BRAINSTORE_INDEX_URI: "gs://{{ .Values.objectStorage.google.brainstoreBucket }}/brainstore/index"
BRAINSTORE_REALTIME_WAL_URI: "gs://{{ .Values.objectStorage.google.brainstoreBucket }}/brainstore/wal"
{{- if eq .Values.brainstore.locksBackend "objectStorage" }}
BRAINSTORE_LOCKS_URI: "gs://{{ .Values.objectStorage.google.brainstoreBucket }}/brainstore/locks"
{{- end }}
{{- if .Values.objectStorage.google.apiBucket }}
BRAINSTORE_RESPONSE_CACHE_URI: "gs://{{ .Values.objectStorage.google.apiBucket }}/response"
BRAINSTORE_CODE_BUNDLE_URI: "gs://{{ .Values.objectStorage.google.apiBucket }}/code-bundle"
{{- end }}
{{- end }}
BRAINSTORE_CONTROL_PLANE_TELEMETRY: {{ .Values.global.controlPlaneTelemetry | quote }}
NO_COLOR: "1"
{{- if .Values.skipPgForBrainstoreObjects }}
BRAINSTORE_ASYNC_SCORING_OBJECTS: {{ .Values.skipPgForBrainstoreObjects | quote }}
BRAINSTORE_LOG_AUTOMATIONS_OBJECTS: {{ .Values.skipPgForBrainstoreObjects | quote }}
{{- end }}
22 changes: 21 additions & 1 deletion braintrust/templates/brainstore-reader-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,38 @@ data:
{{- if eq .Values.brainstore.locksBackend "objectStorage" }}
BRAINSTORE_LOCKS_URI: "az://{{ .Values.objectStorage.azure.brainstoreContainer }}/brainstore/locks"
{{- end }}
{{- if .Values.objectStorage.azure.responseContainer }}
BRAINSTORE_RESPONSE_CACHE_URI: "az://{{ .Values.objectStorage.azure.responseContainer }}"
{{- end }}
{{- if .Values.objectStorage.azure.codeBundleContainer }}
BRAINSTORE_CODE_BUNDLE_URI: "az://{{ .Values.objectStorage.azure.codeBundleContainer }}"
{{- end }}
{{- else if eq .Values.cloud "aws" }}
BRAINSTORE_INDEX_URI: "s3://{{ .Values.objectStorage.aws.brainstoreBucket }}/brainstore/index"
BRAINSTORE_REALTIME_WAL_URI: "s3://{{ .Values.objectStorage.aws.brainstoreBucket }}/brainstore/wal"
{{- if eq .Values.brainstore.locksBackend "objectStorage" }}
BRAINSTORE_LOCKS_URI: "s3://{{ .Values.objectStorage.aws.brainstoreBucket }}/brainstore/locks"
{{- end }}
{{- if .Values.objectStorage.aws.responseBucket }}
BRAINSTORE_RESPONSE_CACHE_URI: "s3://{{ .Values.objectStorage.aws.responseBucket }}"
{{- end }}
{{- if .Values.objectStorage.aws.codeBundleBucket }}
BRAINSTORE_CODE_BUNDLE_URI: "s3://{{ .Values.objectStorage.aws.codeBundleBucket }}"
{{- end }}
{{- else if eq .Values.cloud "google" }}
BRAINSTORE_INDEX_URI: "gs://{{ .Values.objectStorage.google.brainstoreBucket }}/brainstore/index"
BRAINSTORE_REALTIME_WAL_URI: "gs://{{ .Values.objectStorage.google.brainstoreBucket }}/brainstore/wal"
{{- if eq .Values.brainstore.locksBackend "objectStorage" }}
BRAINSTORE_LOCKS_URI: "gs://{{ .Values.objectStorage.google.brainstoreBucket }}/brainstore/locks"
{{- end }}
{{- if .Values.objectStorage.google.apiBucket }}
BRAINSTORE_RESPONSE_CACHE_URI: "gs://{{ .Values.objectStorage.google.apiBucket }}/response"
BRAINSTORE_CODE_BUNDLE_URI: "gs://{{ .Values.objectStorage.google.apiBucket }}/code-bundle"
{{- end }}
{{- end }}
BRAINSTORE_CONTROL_PLANE_TELEMETRY: {{ .Values.global.controlPlaneTelemetry | quote }}
NO_COLOR: "1"
NO_COLOR: "1"
{{- if .Values.skipPgForBrainstoreObjects }}
BRAINSTORE_ASYNC_SCORING_OBJECTS: {{ .Values.skipPgForBrainstoreObjects | quote }}
BRAINSTORE_LOG_AUTOMATIONS_OBJECTS: {{ .Values.skipPgForBrainstoreObjects | quote }}
{{- end }}
Loading
Loading