[CRE] [2/5] Relay DON node handler for confidential relay by nadahalli · Pull Request #21639 · smartcontractkit/chainlink

nadahalli · 2026-03-23T17:00:37Z

Context

Part of #21635 (confidential workflow execution). [2/5] in the series.
Can be reviewed and merged independently.

What this does

Adds a GatewayConnectorHandler that runs on relay DON nodes. It
validates Nitro attestation on incoming enclave requests, then proxies
to VaultDON (secrets_get) or capability DONs (capability_exec).

Each relay node calls the target DON through CRE's standard capability
dispatch (which includes DON-level consensus), receives the aggregated
response, and translates it deterministically before returning to the
gateway. The gateway handler (#21638, [1/5]) aggregates these responses
using F+1 quorum.

Supports multi-PCR validation for enclave pools where each instance
has unique PCR values.

Config: CRE.ConfidentialRelay TOML block (enabled, trustedPCRs,
caRootsPEM) for relay DON node configuration.

Dependencies

None. This PR is self-contained.

github-actions · 2026-03-23T17:00:51Z

👋 nadahalli, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

github-actions · 2026-03-23T17:01:51Z

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

#added For any new functionality added.
#breaking_change For any functionality that requires manual action for the node to boot.
#bugfix For bug fixes.
#changed For any change to the existing functionality.
#db_update For any feature that introduces updates to database schema.
#deprecation_notice For any upcoming deprecation functionality.
#internal For changesets that need to be excluded from the final changelog.
#nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
#removed For any functionality/config that is removed.
#updated For any functionality that is updated.
#wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

github-actions · 2026-03-23T17:02:29Z

✅ No conflicts with other open PRs targeting develop

Copilot

Pull request overview

Risk Rating: HIGH

Adds confidential-relay request handling across the gateway and relay DON nodes as part of confidential workflow execution (#21635), including quorum aggregation on the gateway side and Nitro-attestation validation on the relay node side before proxying to Vault/capability execution.

Changes:

Introduces a new gateway handler that fans out confidential relay requests to relay DON nodes and aggregates node responses to an F+1 quorum.
Introduces a new relay DON node GatewayConnectorHandler that validates Nitro attestations (including multi-PCR pools) and proxies secrets_get/capability_exec to Vault/capabilities.
Updates dependencies to include attestation-related packages (chainlink-common/pkg/teeattestation, hf/nitrite) and bumps fxamacker/cbor.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
go.mod	Bumps/adds dependencies needed for TEE attestation + relay protocol types.
go.sum	Records new/updated module checksums for the dependency changes.
core/services/gateway/handlers/confidentialrelay/handler.go	New gateway-side fanout + quorum aggregation handler with metrics and request lifecycle management.
core/services/gateway/handlers/confidentialrelay/aggregator.go	New quorum aggregator for matching node responses (digest-based) with “quorum unobtainable” detection.
core/services/gateway/handlers/confidentialrelay/handler_test.go	Unit tests covering quorum success/divergence, timeouts, duplicate IDs, fanout failures, and rate-limiting.
core/capabilities/confidentialrelay/handler.go	New relay DON node handler validating Nitro attestation and proxying to Vault/capability execution.
core/capabilities/confidentialrelay/handler_test.go	Unit tests for relay handler behavior (attestation, capability exec, unsupported methods, invalid params).
core/capabilities/confidentialrelay/service.go	Lifecycle wrapper that defers handler creation until the gateway connector is available.

Copilot · 2026-03-23T17:07:25Z

core/capabilities/confidentialrelay/handler.go

+func (h *Handler) handleSecretsGet(ctx context.Context, gatewayID string, req *jsonrpc.Request[json.RawMessage]) *jsonrpc.Response[json.RawMessage] {
+	var params confidentialrelaytypes.SecretsRequestParams
+	if err := json.Unmarshal(*req.Params, &params); err != nil {
+		return h.errorResponse(ctx, gatewayID, req, jsonrpc.ErrInvalidParams, err)
+	}


handleSecretsGet dereferences *req.Params without checking for req.Params == nil. A JSON-RPC request can omit params, which would panic here. Add a nil check and return a JSON-RPC invalid request/invalid params error response instead of crashing the handler.

Copilot · 2026-03-23T17:07:26Z

core/capabilities/confidentialrelay/handler.go

+}
+
+func (h *Handler) handleCapabilityExecute(ctx context.Context, gatewayID string, req *jsonrpc.Request[json.RawMessage]) *jsonrpc.Response[json.RawMessage] {
+	var params confidentialrelaytypes.CapabilityRequestParams


handleCapabilityExecute dereferences *req.Params without checking for req.Params == nil, which can panic for JSON-RPC requests that omit params. Add a nil check and return an appropriate JSON-RPC invalid request/invalid params response.

Suggested change

var params confidentialrelaytypes.CapabilityRequestParams

var params confidentialrelaytypes.CapabilityRequestParams

if req.Params == nil {

return h.errorResponse(ctx, gatewayID, req, jsonrpc.ErrInvalidParams, fmt.Errorf("missing params"))

}

Copilot · 2026-03-23T17:07:26Z

core/capabilities/confidentialrelay/handler.go

+	// Normalize owner to EIP-55 checksum format, matching how secrets are stored.
+	normalizedOwner := common.HexToAddress(params.Owner).Hex()
+


common.HexToAddress(params.Owner) will silently coerce invalid/empty owner strings to the zero address. If params.Owner is malformed, this could query Vault under an unintended owner without surfacing an error. Consider validating params.Owner first (e.g., common.IsHexAddress) and returning an ErrInvalidParams response when it’s not a valid EVM address.

Copilot · 2026-03-23T17:07:26Z

core/capabilities/confidentialrelay/handler.go

+	h.metrics.requestSuccess.Add(ctx, 1, metric.WithAttributes(
+		attribute.String("gateway_id", gatewayID),
+	))


HandleGatewayMessage increments requestSuccess unconditionally after sending the response, even when the response contains a JSON-RPC error (and errorResponse already increments requestInternalError). This will skew success/error rates in telemetry. Consider incrementing success only when response.Error == nil (and using the error counter otherwise).

Suggested change

h.metrics.requestSuccess.Add(ctx, 1, metric.WithAttributes(

attribute.String("gateway_id", gatewayID),

))

if response != nil && response.Error == nil {

h.metrics.requestSuccess.Add(ctx, 1, metric.WithAttributes(

attribute.String("gateway_id", gatewayID),

))

}

Copilot · 2026-03-23T17:07:27Z

core/services/gateway/handlers/confidentialrelay/handler.go

+package confidentialrelay
+
+import (
+	"context"
+	"encoding/json"


PR title/description emphasize the relay DON node handler, but this PR also introduces a new gateway-side handler (core/services/gateway/handlers/confidentialrelay/*) that fans out and aggregates responses. If that’s intentional, consider updating the PR description (or splitting) so reviewers don’t miss the gateway-surface changes.

Fixed. The gateway handler files were erroneously included in this PR. They belong to PR #21638 [1/5]. Removed.

Copilot · 2026-03-23T17:07:27Z

core/services/gateway/handlers/confidentialrelay/handler.go

+		l.Debugw("aggregating responses, waiting for other nodes...", "error", err)
+		return nil
+	case err != nil:
+		l.Error("quorum unobtainable, returning response to user...", "error", err, "responses", maps.Values(ar.responses))


maps.Values(ar.responses) reads the underlying map without holding ar.mu. Since ar.responses is mutated under addResponseForNode, this log line can trigger a data race / concurrent map iteration panic under load. Use copiedResponses := ar.copiedResponses() (or lock around the read) and log the copied map/values instead.

Suggested change

l.Error("quorum unobtainable, returning response to user...", "error", err, "responses", maps.Values(ar.responses))

l.Error("quorum unobtainable, returning response to user...", "error", err, "responses", copiedResponses)

This file was erroneously included and has been removed from this PR. Tracked in #21638.

Copilot · 2026-03-23T17:07:27Z

core/services/gateway/handlers/confidentialrelay/handler.go

+func (h *handler) HandleJSONRPCUserMessage(ctx context.Context, req jsonrpc.Request[json.RawMessage], callback gwhandlers.Callback) error {
+	if req.ID == "" {
+		return errors.New("request ID cannot be empty")
+	}
+	if len(req.ID) > 200 {
+		return errors.New("request ID is too long: " + strconv.Itoa(len(req.ID)) + ". max is 200 characters")
+	}
+
+	l := logger.With(h.lggr, "method", req.Method, "requestID", req.ID)
+	l.Debugw("handling confidential relay request")
+
+	ar, err := h.newActiveRequest(req, callback)
+	if err != nil {
+		return err
+	}
+
+	return h.fanOutToNodes(ctx, l, ar)


HandleJSONRPCUserMessage forwards any req.Method without validation and doesn’t guard against req.Params == nil. Other gateway handlers validate supported methods and reject missing params (e.g., core/services/gateway/handlers/vault/handler.go:336-396 and core/services/gateway/handlers/capabilities/v2/http_trigger_handler.go:127-131). Consider failing fast here with UnsupportedMethodError/InvalidParamsError to avoid panics later (notably errorResponse dereferences *req.Params).

This file was erroneously included and has been removed from this PR. Tracked in #21638.

github-actions · 2026-03-23T17:20:01Z

CORA - Pending Reviewers

Codeowners Entry	Overall	Num Files	Owners
`*`	💬	8	@smartcontractkit/foundations, @smartcontractkit/core
`/core/capabilities/`	💬	3	@smartcontractkit/keystone, @smartcontractkit/capabilities-team
`/core/web/resolver`	💬	3	@smartcontractkit/foundations, @smartcontractkit/core
`go.md`	💬	1	@smartcontractkit/core, @smartcontractkit/foundations
`go.mod`	💬	6	@smartcontractkit/core, @smartcontractkit/foundations
`go.sum`	💬	6	@smartcontractkit/core, @smartcontractkit/foundations
`integration-tests/go.mod`	💬	1	@smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations
`integration-tests/go.sum`	💬	1	@smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations

For more details, see the full review summary.

trunk-io · 2026-03-23T20:12:55Z

Failed Test	Failure Summary	Logs
`TestScripts/node/validate`	The test named 'TestScripts/node/validate' failed during execution.	Logs ↗︎
`TestScripts/node/validate`	The test 'TestScripts/node/validate' failed without providing specific error details.	Logs ↗︎
`TestConfigDocs`	The test failed because the actual documentation content did not match the expected documentation string.	Logs ↗︎
`TestConfigDocs`	The test failed because the actual documentation content did not match the expected documentation string.	Logs ↗︎

... and 24 more

View Full Report ↗︎ ⋅ Docs

vreff · 2026-03-24T11:31:23Z

core/capabilities/confidentialrelay/handler.go

+	lggr             logger.Logger
+	metrics          *handlerMetrics
+
+	// validateAttestation validates Nitro attestation documents.


nit: nothing nitro specific about this function definition

Fixed. Replaced "Nitro" with "TEE" in the type and handler comments.

vreff · 2026-03-24T11:37:42Z

core/capabilities/confidentialrelay/service.go

+	trustedPCRs []byte,
+	caRootsPEM string,


why do we need trustedPCRs as a parameter? They should be retrievable from the capRegistry parameter. The way executor.go does things is:

Parse the configs from the registry

Extract measurements from the list of enclaves.

Any incoming request matching those measurements is accepted.

If this is already being done outside of this logic, why not bring it within this component? I doubt the PCRs are being used anywhere else.

Addressed. Both TrustedPCRs and CARootsPEM are now read from the capabilities registry per-request via DONsForCapability + ConfigForCapability, same pattern as CC's EnsureFreshEnclaves in executor.go. The handler looks up the confidential-workflows capability config, extracts TrustedValues and CARootsPEM from the EnclavesList, and uses them for attestation validation.

CRE.ConfidentialRelay TOML config is now just Enabled (a kill switch for relay participation).

vreff · 2026-03-24T11:39:39Z

core/capabilities/confidentialrelay/handler.go

+	// Each enclave instance has different PCR values (WireGuard keys baked
+	// per CID), so trustedPCRs may be a JSON array of PCR objects. Try each
+	// set and succeed if any match, same as pool.go's
+	// validateAttestationAgainstMultipleMeasurements.
+	var pcrArray []json.RawMessage
+	if json.Unmarshal(h.trustedPCRs, &pcrArray) == nil && len(pcrArray) > 0 {
+		var validationErr error
+		for _, pcrs := range pcrArray {
+			err := h.validateAttestation(attestationBytes, hash, pcrs, h.caRootsPEM)
+			if err == nil {
+				return nil
+			}
+			validationErr = errors.Join(validationErr, err)
+		}
+		return fmt.Errorf("no trusted PCR set matched: %w", validationErr)
+	}


Feel like this is a bit messy for unmarshalling to be within this logic every time rather than within the constructor.

This is addressed. The trustedPCRs parameter is gone. Measurements are now read from the capabilities registry per-request via getEnclaveAttestationConfig, which returns structured data directly. No JSON unmarshalling in the hot path.

vreff · 2026-03-25T15:18:21Z

core/capabilities/confidentialrelay/handler.go

+
+// enclaveEntry mirrors the enclave config shape stored in the capabilities
+// registry. Only the fields needed for attestation validation are included.
+type enclaveEntry struct {


This doesn't match our actual types:

type Enclave struct {
EnclaveID [32]byte json:"enclaveId"
EnclaveURL string json:"enclaveURL"
TrustedValues [][]byte json:"trustedValues" // Array of trusted measurements the client accepts.
EnclaveType EnclaveType json:"enclaveType"
EnclaveAuthHeader string json:"enclaveAuthHeader"
EnclaveExtraData []byte json:"enclaveExtraData"
}

this whole CARootsPem thing is weird. IMO, we should not include this field in our system. I think we'd be able to tell the attestation type from the attestation itself and use the right root certificate.

At any rate, we don't have any reason to not just hardcode it to the Nitro one within chainlink common, for now.

If we do that, we should just get rid of CARootsPem everywhere in chainlink core, and assume the attestationValidatorFunc knows what to do.

Agreed. CARootsPEM should not be in the system. I've updated the teeattestation package in chainlink-common to remove it from the production API:

smartcontractkit/chainlink-common@c7b8c137

ValidateAttestation now takes 3 args (always uses the hardcoded AWS Nitro root). ValidateAttestationWithRoots exists for tests with fake enclaves. Once chainlink-common PR #1899 merges, I'll bump common here and drop the caRootsPEM plumbing from the handler.

Re: the type mismatch with CC's Enclave struct, the handler only needs TrustedValues and CARootsPEM (soon just TrustedValues) from the enclave config. Mirroring unused fields (EnclaveID, EnclaveURL, etc.) would be dead code. Just having TrustedValues should be ok, no?

Changed code now.

vreff · 2026-03-25T15:19:05Z

LGTM apart from getting rid of the root ca pem stuff.

MStreet3 · 2026-03-31T17:07:06Z

core/capabilities/confidentialrelay/handler.go

+	}, nil
+}
+
+type gatewayConnector interface {


nit why is this a private interface yet used in the constructor. I get you may want to slim it down but clients won't be able to inspect the constructor param.

MStreet3 · 2026-03-31T17:10:38Z

core/capabilities/confidentialrelay/handler.go

+		return h.errorResponse(ctx, gatewayID, req, jsonrpc.ErrInternal, fmt.Errorf("failed to get local node: %w", err))
+	}
+
+	capResp, err := vaultCap.Execute(ctx, capabilities.CapabilityRequest{


No metrics on latency or count of executions?

+1 on metrics

Added count and latency metrics for requests.

george-dorin · 2026-04-01T11:59:33Z

core/capabilities/confidentialrelay/handler.go

+		},
+	})
+	if err != nil {
+		return h.errorResponse(ctx, gatewayID, req, jsonrpc.ErrInternal, fmt.Errorf("vault execute failed: %w", err))


Is there any chance the err message could leak sensitive information?

You never know. Changed it to "internal error" for safety. Logging the error though. I think we are careful enough not to send sensitive error messages from upstream. So, logging should be ok.

Add GatewayConnectorHandler that validates Nitro attestation and proxies enclave requests to VaultDON (secrets_get) and capability DONs (capability_exec). Supports multi-PCR validation for enclave pools. Part of #21635

Replace the static trustedPCRs TOML parameter with a per-request lookup from the capabilities registry. The handler reads the confidential-workflows capability config to extract enclave TrustedValues, same pattern as CC's EnsureFreshEnclaves. Remove TrustedPCRs from CREConfidentialRelay config interface, ConfidentialRelayConfig TOML struct, and config_cre.go.

CARootsPEM is stored per-enclave in the same cap registry config as TrustedValues. Read both in getEnclaveAttestationConfig, eliminating the last TOML field besides Enabled from CRE.ConfidentialRelay.

ValidateAttestation in chainlink-common now always uses the hardcoded AWS Nitro root cert (c7b8c137). Drop the caRootsPEM parameter from the handler's attestation validator func, getEnclaveAttestationConfig, and the enclaveEntry struct.

(cherry picked from commit 16e4fc9)

cl-sonarqube-production · 2026-04-02T16:02:02Z

Quality Gate passed

Issues
0 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

Copilot AI review requested due to automatic review settings March 23, 2026 17:00

nadahalli requested review from a team as code owners March 23, 2026 17:00

product-security-plaid-production bot requested review from MStreet3 and george-dorin March 23, 2026 17:00

Copilot started reviewing on behalf of nadahalli March 23, 2026 17:01 View session

This was referenced Mar 23, 2026

[CRE] [5/5] Wire confidential workflow execution into CRE #21642

Open

[CRE] Confidential workflow execution #21635

Open

Copilot AI reviewed Mar 23, 2026

View reviewed changes

nadahalli requested a review from a team as a code owner March 23, 2026 17:19

product-security-plaid-production bot requested a review from skudasov March 23, 2026 17:19

nadahalli force-pushed the tejaswi/cw-2-relay-handler branch 2 times, most recently from ef1dc22 to e9eb648 Compare March 23, 2026 19:56

vreff reviewed Mar 24, 2026

View reviewed changes

nadahalli mentioned this pull request Mar 25, 2026

[CRE] [1/5] Gateway handler for confidential relay #21638

Open

vreff reviewed Mar 25, 2026

View reviewed changes

vreff previously approved these changes Mar 26, 2026

View reviewed changes

nadahalli dismissed vreff’s stale review via 70acaf1 March 26, 2026 15:06

nadahalli force-pushed the tejaswi/cw-2-relay-handler branch from c76aea8 to 70acaf1 Compare March 26, 2026 15:06

vreff previously approved these changes Mar 30, 2026

View reviewed changes

nadahalli dismissed vreff’s stale review via e229c3f March 31, 2026 16:24

skudasov previously approved these changes Mar 31, 2026

View reviewed changes

MStreet3 reviewed Mar 31, 2026

View reviewed changes

george-dorin reviewed Apr 1, 2026

View reviewed changes

nadahalli dismissed skudasov’s stale review via 9aafa80 April 1, 2026 12:03

skudasov self-requested a review April 1, 2026 22:18

skudasov previously approved these changes Apr 1, 2026

View reviewed changes

nadahalli dismissed skudasov’s stale review via cb65a31 April 2, 2026 14:00

nadahalli added 11 commits April 2, 2026 16:09

[CRE] [2/5] Relay DON node handler for confidential relay

fa65a23

Add GatewayConnectorHandler that validates Nitro attestation and proxies enclave requests to VaultDON (secrets_get) and capability DONs (capability_exec). Supports multi-PCR validation for enclave pools. Part of #21635

Add ConfidentialRelay config for relay DON handler

c302c92

Remove Nitro-specific wording from attestation comments

42a9798

Also read CARootsPEM from cap registry, remove from TOML config

452ca83

CARootsPEM is stored per-enclave in the same cap registry config as TrustedValues. Read both in getEnclaveAttestationConfig, eliminating the last TOML field besides Enabled from CRE.ConfidentialRelay.

Remove CARootsPEM from relay handler

1b80ace

ValidateAttestation in chainlink-common now always uses the hardcoded AWS Nitro root cert (c7b8c137). Drop the caRootsPEM parameter from the handler's attestation validator func, getEnclaveAttestationConfig, and the enclaveEntry struct.

Relay handler: use CARootsPEM from registry for fake enclave attestation

d9939f7

forward relay metadata for remote capability execution

3f16bf6

(cherry picked from commit 16e4fc9)

add relay request metrics and use exported connector type

51a6276

sanitize internal relay error messages

0112f40

bump chainlink-common and sync relay config snapshots

00a16e6

nadahalli force-pushed the tejaswi/cw-2-relay-handler branch from cb65a31 to 00a16e6 Compare April 2, 2026 14:12

Fix confidential relay CI fallout

b84b85e

vreff approved these changes Apr 2, 2026

View reviewed changes

		// Normalize owner to EIP-55 checksum format, matching how secrets are stored.
		normalizedOwner := common.HexToAddress(params.Owner).Hex()

	l.Error("quorum unobtainable, returning response to user...", "error", err, "responses", maps.Values(ar.responses))
	l.Error("quorum unobtainable, returning response to user...", "error", err, "responses", copiedResponses)

Conversation

nadahalli commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

What this does

Dependencies

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

github-actions bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CORA - Pending Reviewers

Uh oh!

trunk-io bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nadahalli Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

nadahalli commented Mar 23, 2026 •

edited

Loading

github-actions bot commented Mar 23, 2026 •

edited

Loading

github-actions bot commented Mar 23, 2026 •

edited

Loading

trunk-io bot commented Mar 23, 2026 •

edited

Loading

nadahalli Mar 26, 2026 •

edited

Loading