Stream Deduplication

[meling]

Goal: enforce one steady-state stream per known peer pair.

• Define deterministic ownership for which side keeps the outbound stream.
• Ensure both directions still work over the retained stream.
• Handle reconnects without reintroducing duplicate steady-state streams.
• Add tests to confirm methods can be invoked on the configuration after potential race

Acceptance Test Matrix

• Exactly one active stream per known peer pair after stabilization.
• Quorum and multicast behavior remains correct with deduplicated streams.
• Reconnect behavior remains stable and race-free.

[meling]

Stream Deduplication Design Options

This document explores options for stream deduplication in symmetric Gorums configurations and outlines an experimental setup to measure the overhead of maintaining one versus two streams between nodes.

Design Options for Stream Deduplication

The core challenge in a symmetric system is that any node might initiate an RPC before a connection is established. To optimize resources, we want nodes to share a single, bidirectional stream. However, if Node A and Node B dial each other at the exact same moment (a race), we could end up with two parallel streams.

Below is an exploration of the three proposed designs to handle this.

(A) Always use two streams (Current Solution)

In this model, every node pair (A, B) maintains two distinct unidirectional (or functionally unidirectional) streams: A → B for A's requests, and B → A for B's requests.

• Implementation: This is the current Gorums behavior. A dials B when it needs to send an RPC, and B dials A when it needs to send an RPC.
• Pros:
  • Extreme simplicity.
  • No complex distributed state or race conditions to mitigate.
  • No blocking or coordination needed during connection establishment.
• Cons:
  • Doubles the number of TCP connections ($N \times (N-1)$ total connections in the system instead of $N \times (N-1) / 2$).
  • Doubles the number of HTTP/2 keep-alives and associated goroutines (gRPC spawns multiple goroutines per connection).
  • High memory footprint at scale.

(B) Shared stream, but keep both alive on race

In this model, nodes attempt to share a stream. Node A dials B, and B registers A's incoming stream and uses it for outbound RPCs back to A. However, if a dial race occurs and two streams are established, the system simply accepts the inefficiency and keeps both alive.

• Implementation:
  • When Node A receives an inbound connection from B via the InboundManager, it checks if it already has an outbound connection to B.
  • If it doesn't, it uses B's inbound stream as its Channel to B.
  • If a race occurs (A dials B, B dials A concurrently), both nodes will register an outbound stream and an inbound stream.
  • Resolution: Both nodes just keep their initiated (outbound) stream as the primary way to send requests, or we can designate the stream initiated by the node with the higher NodeID as the primary for both. Neither stream is closed.
• Pros:
  • Solves the $2\times$ overhead in 99.9% of cases (races only happen under highly concurrent startups).
  • Avoids the extreme complexity of safely closing an in-use stream without dropping buffered messages.
• Cons:
  • System state is non-deterministic (some pairs have 1 stream, some have 2).
  • Requires routing logic in the Channel to prioritize which stream to use if two are active.

(C) Shared stream, safely close one on race

In this model, the system strictly guarantees a single shared stream between any node pair. If a race occurs, one stream is deterministically chosen to be kept, and the other is safely shut down.

• Implementation:
  • Tie-breaker: When A and B detect a double-stream, they compare IDs (e.g., node with ID > peer ID keeps its outbound stream). Let's say A > B. A keeps its outbound stream (A → B) and B keeps its inbound stream (A → B).
  • Safe Closure: B must close its outbound stream (B → A). However, B might have already queued or sent RPCs on this stream! If B just closes the stream, those RPCs might be lost in flight.
  • Coordination Protocol: To safely close, B cannot just severe the connection. B would need to send a FlushAndClose control message to A on the doomed stream, wait for an acknowledgment, and simultaneously redirect all new outgoing RPCs to the kept stream. A must process all messages on the doomed stream up to the FlushAndClose marker.
• Pros:
  • Strictly minimal resource usage (always exactly 1 stream per pair).
  • Clean, deterministic system state.
• Cons:
  • Extremely complex to implement correctly without message loss or deadlocks.
  • Introduces an application-level control protocol inside the Gorums stream multiplexer.
  • Connection establishment latency might increase due to lock contention or coordination blocks.

---

Benchmark & Experimental Plan

To confidently choose between (A) and (B)/(C), we need to measure the actual cost of a gRPC stream in Gorums.

Scenario: The StreamOverhead Benchmark

We will create a benchmark/test suite that spins up $N$ nodes in a completely symmetric configuration. We will expose a configuration toggle: WithForcedDualStreams(bool).

Metrics to Capture:

1. Idle Resource Overhead:
   • Goroutine Count: runtime.NumGoroutine() at steady idle state.
   • Memory Allocation: Measured via runtime.ReadMemStats (specifically looking at Alloc and Sys to gauge the memory footprint of the gRPC transport buffers and keep-alive timers).
2. Throughput & Latency (Active Overhead):
   • Run a barrage of Multicast messages across all nodes.
   • Measure CPU usage/time (testing.B) and latency percentiles.

Test Matrix:

• Sizes: $N = 10, 50, 100$ nodes.
• Mode 1: Strict 1-Stream (Simulated perfectly by starting nodes sequentially so no races occur, using a shared stream).
• Mode 2: Strict 2-Stream (Design A).

Expected Outcome

• If gRPC's memory and goroutine overhead for $N=100$ (10,000 connections vs 5,000 connections) causes significant GC pressure or memory bloat, it heavily justifies moving away from (A).
• If 1-Stream proves highly beneficial, Design (B) is the most pragmatic approach, offering the benefits without the devastating complexity of Design (C)'s safe-closure coordination.