Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,29 @@
-run='^TestSpamoorSuite$/^TestDeFiSimulation$' -v -timeout=15m \
./benchmark/ --evm-binary=../../../build/evm

# TODO: wire up to publish results once additional tests are in place.
state-pressure-benchmark:
name: State Pressure Benchmark
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Go
uses: actions/setup-go@4b73464bb391d4059bd26b0524d20df3927bd417 # v6.3.0
with:
go-version-file: ./go.mod
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Install just
uses: extractions/setup-just@v3

Check warning

Code scanning / CodeQL

Unpinned tag for a non-immutable Action in workflow Medium

Unpinned 3rd party Action 'Benchmarks' step
Uses Step
uses 'extractions/setup-just' with ref 'v3', not a pinned commit hash
Comment on lines +138 to +139
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Get the commit SHA for extractions/setup-just v3 tag
curl -s "https://api.github.com/repos/extractions/setup-just/git/refs/tags/v3" | jq -r '.object.sha'

Repository: evstack/ev-node

Length of output: 63


🏁 Script executed:

#!/bin/bash

# Try alternative method to get the commit SHA for the v3 tag
# Using the commits API instead
curl -s "https://api.github.com/repos/extractions/setup-just/releases/tags/v3" | jq -r '.target_commitish'

echo "---"

# Also try getting commit info directly
curl -s "https://api.github.com/repos/extractions/setup-just/git/refs/tags/v3" | jq '.'

Repository: evstack/ev-node

Length of output: 448


🏁 Script executed:

#!/bin/bash

# Check if the repository exists and list recent releases
curl -s "https://api.github.com/repos/extractions/setup-just" | jq -r '.name, .full_name, .archived'

echo "---"

# Get all releases
curl -s "https://api.github.com/repos/extractions/setup-just/releases" | jq -r '.[0:3] | .[] | "\(.tag_name): \(.target_commitish)"'

Repository: evstack/ev-node

Length of output: 108


🏁 Script executed:

#!/bin/bash

# Check the benchmark.yml file around lines 115-116
sed -n '110,120p' .github/workflows/benchmark.yml

Repository: evstack/ev-node

Length of output: 501


Pin setup-just action to a commit hash.

The extractions/setup-just@v3 action is not pinned to a specific commit hash, unlike other actions in this workflow (e.g., actions/setup-go, docker/setup-buildx-action). Pinning to a commit hash improves supply chain security.

🔒 Proposed fix to pin the action
       - name: Install just
-        uses: extractions/setup-just@v3
+        uses: extractions/setup-just@f8a3cce218d9f83db3a2ecd90e41ac3de6cdfd9b # v3
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Install just
uses: extractions/setup-just@v3
- name: Install just
uses: extractions/setup-just@f8a3cce218d9f83db3a2ecd90e41ac3de6cdfd9b # v3
🧰 Tools
🪛 GitHub Check: CodeQL

[warning] 116-116: Unpinned tag for a non-immutable Action in workflow
Unpinned 3rd party Action 'Benchmarks' step Uses Step uses 'extractions/setup-just' with ref 'v3', not a pinned commit hash

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/benchmark.yml around lines 115 - 116, Replace the unpinned
GitHub Action reference uses: extractions/setup-just@v3 with the same action
pinned to a specific commit SHA so the workflow uses a fixed revision; locate
the line referencing the setup-just action (uses: extractions/setup-just@v3) and
update it to the commit-pinned form (the action's repository@<commit-sha>)
matching how other actions are pinned in this workflow.

- name: Build binaries
run: just build-evm build-da
- name: Run state pressure test
run: |
cd test/e2e && go test -tags evm \
-run='^TestSpamoorSuite$/^TestStatePressure$' -v -timeout=15m \
./benchmark/ --evm-binary=../../../build/evm

# single job to push all results to gh-pages sequentially, avoiding race conditions
publish-benchmarks:
name: Publish Benchmark Results
Expand Down
114 changes: 114 additions & 0 deletions test/e2e/benchmark/spamoor_state_pressure_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
//go:build evm

package benchmark

import (
"context"
"fmt"
"time"

"github.com/celestiaorg/tastora/framework/docker/evstack/spamoor"
)

// TestStatePressure measures throughput under maximum storage write pressure.
// Each tx maximizes SSTORE operations, creating rapid state growth that stresses
// the state trie and disk I/O.
//
// Shares system configuration with TestERC20Throughput (100ms blocks, 100M gas,
// 25ms scrape) so results are directly comparable. The gap between
// TestEVMComputeCeiling and this test isolates state root + storage I/O cost.
//
// Primary metrics: MGas/s.
// Diagnostic metrics: Engine.NewPayload latency, ev-node overhead %.
func (s *SpamoorSuite) TestStatePressure() {
cfg := newBenchConfig("ev-node-state-pressure")

t := s.T()
ctx := t.Context()
cfg.log(t)
w := newResultWriter(t, "StatePressure")
defer w.flush()

e := s.setupEnv(cfg)

storageSpamConfig := map[string]any{
"throughput": cfg.Throughput,
"total_count": cfg.CountPerSpammer,
"gas_units_to_burn": cfg.GasUnitsToBurn,
"max_pending": 50000,
"max_wallets": cfg.MaxWallets,
"base_fee": 20,
"tip_fee": 2,
"refill_amount": "5000000000000000000", // 5 ETH
"refill_balance": "2000000000000000000", // 2 ETH
"refill_interval": 600,
}

s.Require().NoError(deleteAllSpammers(e.spamoorAPI), "failed to delete stale spammers")

var spammerIDs []int
for i := range cfg.NumSpammers {
name := fmt.Sprintf("bench-storage-%d", i)
id, err := e.spamoorAPI.CreateSpammer(name, spamoor.ScenarioStorageSpam, storageSpamConfig, true)
s.Require().NoError(err, "failed to create spammer %s", name)
spammerIDs = append(spammerIDs, id)
t.Cleanup(func() { _ = e.spamoorAPI.DeleteSpammer(id) })
}

// allow spamoor time to initialise spammer goroutines before polling status
time.Sleep(3 * time.Second)
requireSpammersRunning(t, e.spamoorAPI, spammerIDs)

// wait for wallet funding to finish before recording start block
pollSentTotal := func() (float64, error) {
metrics, mErr := e.spamoorAPI.GetMetrics()
if mErr != nil {
return 0, mErr
}
return sumCounter(metrics["spamoor_transactions_sent_total"]), nil
}
waitForMetricTarget(t, "spamoor_transactions_sent_total (warmup)", pollSentTotal, float64(cfg.WarmupTxs), cfg.WaitTimeout)

// reset trace window to exclude warmup spans
e.traces.resetStartTime()

startHeader, err := e.ethClient.HeaderByNumber(ctx, nil)
s.Require().NoError(err, "failed to get start block header")
startBlock := startHeader.Number.Uint64()
loadStart := time.Now()
t.Logf("start block: %d (after warmup)", startBlock)

// wait for all transactions to be sent
waitForMetricTarget(t, "spamoor_transactions_sent_total", pollSentTotal, float64(cfg.totalCount()), cfg.WaitTimeout)

// wait for pending txs to drain
drainCtx, drainCancel := context.WithTimeout(ctx, 30*time.Second)
defer drainCancel()
if err := waitForDrain(drainCtx, t.Logf, e.ethClient, 10); err != nil {
t.Logf("warning: %v", err)
}
wallClock := time.Since(loadStart)

endHeader, err := e.ethClient.HeaderByNumber(ctx, nil)
s.Require().NoError(err, "failed to get end block header")
endBlock := endHeader.Number.Uint64()
t.Logf("end block: %d (range %d blocks)", endBlock, endBlock-startBlock)

// collect block-level gas/tx metrics
bm, err := collectBlockMetrics(ctx, e.ethClient, startBlock, endBlock)
s.Require().NoError(err, "failed to collect block metrics")

traces := s.collectTraces(e, cfg.ServiceName)

result := newBenchmarkResult("StatePressure", bm, traces)
s.Require().Greater(result.summary.SteadyState, time.Duration(0), "expected non-zero steady-state duration")
result.log(t, wallClock)
w.addEntries(result.entries())

metrics, mErr := e.spamoorAPI.GetMetrics()
s.Require().NoError(mErr, "failed to get final metrics")
sent := sumCounter(metrics["spamoor_transactions_sent_total"])
failed := sumCounter(metrics["spamoor_transactions_failed_total"])
s.Require().Greater(sent, float64(0), "at least one transaction should have been sent")
s.Require().Zero(failed, "no transactions should have failed")
}
Loading