Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .agents/skills/obol-stack-dev/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
name: obol-stack-dev
description: Obol Stack development, testing, and LLM routing validation through LiteLLM. Use when developing, testing, or validating inference paths (Ollama, Anthropic, OpenAI) through the LiteLLM gateway, writing integration tests, or working with obol CLI wrappers.
description: Obol Stack development, testing, and validation. Covers LLM routing through LiteLLM, x402 payment flow (sell/buy), BDD integration tests (Gherkin/godog), ERC-8004 registration, and obol CLI wrappers.
metadata:
version: "2.0.0"
domain: infrastructure
triggers: obol, litellm, openclaw, inference, integration test, model routing, smart routing, LLM proxy, provider setup
triggers: obol, litellm, openclaw, inference, integration test, model routing, smart routing, LLM proxy, provider setup, x402, sell, buy, BDD, gherkin, payment, monetize
role: specialist
scope: development-and-testing
output-format: code-and-commands
Expand All @@ -20,8 +20,10 @@ Complete guide for developing, testing, and validating the Obol Stack's LLM rout
- Setting up the Obol Stack development environment
- Testing LLM inference through LiteLLM (Ollama, Anthropic, OpenAI)
- Writing or running integration tests for OpenClaw instances
- Running BDD integration tests for the x402 sell→discover→buy payment flow
- Debugging model routing issues (401s, 500s, provider misconfig)
- Understanding the 2-tier LLM architecture (LiteLLM gateway + per-instance config)
- Testing x402 payment gating, ERC-8004 registration, OASF metadata
- Deploying and validating OpenClaw instances with different providers
- Working with the `obol` CLI wrappers (kubectl, helm, helmfile, k9s)

Expand Down
75 changes: 75 additions & 0 deletions .agents/skills/obol-stack-dev/references/integration-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,81 @@ func TestIntegration_MyTest(t *testing.T) {
5. Deploy via `obol openclaw sync`
6. Model name format: `openai/<model-id>` (always openai/ prefix through LiteLLM)

## BDD Integration Tests (x402 Payment Flow)

A separate BDD test suite validates the full sell→discover→buy payment flow using Gherkin scenarios and godog. These tests live in `internal/x402/` and follow the **real user journey** — no kubectl shortcuts.

### Running BDD Tests

```bash
# Against existing cluster (fast, ~2min)
export OBOL_DEVELOPMENT=true
export OBOL_CONFIG_DIR=$(pwd)/.workspace/config
export OBOL_BIN_DIR=$(pwd)/.workspace/bin
export OBOL_DATA_DIR=$(pwd)/.workspace/data
export OBOL_INTEGRATION_SKIP_BOOTSTRAP=true
export OBOL_TEST_MODEL=qwen3.5:9b

go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/

# Full bootstrap from scratch (~15min, creates cluster)
go test -tags integration -v -run TestBDDIntegration -timeout 20m ./internal/x402/
```

### BDD Scenarios (7 total, 77 steps)

| Scenario | What it validates |
|----------|------------------|
| Operator sells inference via CLI + agent reconciles | `obol sell http` → CRD → 6-stage reconciliation |
| Unpaid request returns 402 | x402 payment gate |
| Paid request returns real inference | EIP-712 → verify → LiteLLM → Ollama |
| Discovery-to-payment cycle | Parse 402 → sign → pay → 200 |
| Paid request through Cloudflare tunnel | Full flow via tunnel |
| Agent discovers service through tunnel | `.well-known` → x402Support → OASF skills/domains → probe 402 |
| Operator deletes + cleanup | CR + pricing route removed |

### BDD Test Files

| File | Role |
|------|------|
| `internal/x402/features/integration_payment_flow.feature` | Gherkin scenarios |
| `internal/x402/bdd_integration_test.go` | TestMain bootstrap (builds binary, stack init/up, agent init, sell) |
| `internal/x402/bdd_integration_steps_test.go` | Step definitions (Anvil, facilitator, payment signing) |

### BDD TestMain Bootstrap (Full Mode)

```
1. go build ./cmd/obol
2. obol stack init + up
3. obol model setup (Anthropic/OpenAI/Ollama)
4. obol sell pricing --wallet ... --chain base-sepolia
5. obol agent init (deploys agent + RBAC + monetize skill)
6. obol sell http bdd-test --upstream litellm --port 4000 ...
7. Wait for agent reconciliation (6 stages → Ready)
8. Restart x402-verifier
```

### Skip-Bootstrap Mode

Set `OBOL_INTEGRATION_SKIP_BOOTSTRAP=true` to use an existing cluster. Requires:
- ServiceOffer `bdd-test` in `llm` namespace (Ready)
- x402-verifier running
- LiteLLM running with at least one model
- Ollama running (for inference)

### Per-Scenario Infrastructure

Each scenario gets a fresh:
- Anvil fork (Base Sepolia, on random port)
- Mock facilitator (accepts all payments)
- x402-verifier patched to use the scenario's facilitator

### Key Dependency

```bash
go get github.com/cucumber/godog@v0.15.1
```

## Timing and Timeouts

| Operation | Typical Time | Timeout |
Expand Down
14 changes: 14 additions & 0 deletions cmd/obol/sell.go
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,14 @@ Examples:
Name: "register-image",
Usage: "Agent image URL for ERC-8004 registration",
},
&cli.StringSliceFlag{
Name: "register-skills",
Usage: "OASF skills for discovery (e.g. natural_language_processing/text_generation)",
},
&cli.StringSliceFlag{
Name: "register-domains",
Usage: "OASF domains for discovery (e.g. technology/artificial_intelligence)",
},
},
Action: func(ctx context.Context, cmd *cli.Command) error {
if cmd.NArg() == 0 {
Expand Down Expand Up @@ -339,6 +347,12 @@ Examples:
if img := cmd.String("register-image"); img != "" {
reg["image"] = img
}
if skills := cmd.StringSlice("register-skills"); len(skills) > 0 {
reg["skills"] = skills
}
if domains := cmd.StringSlice("register-domains"); len(domains) > 0 {
reg["domains"] = domains
}
spec["registration"] = reg
}

Expand Down
8 changes: 8 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ require (
github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
github.com/crate-crypto/go-eth-kzg v1.4.0 // indirect
github.com/crate-crypto/go-ipa v0.0.0-20240724233137-53bbb0ceb27a // indirect
github.com/cucumber/gherkin/go/v26 v26.2.0 // indirect
github.com/cucumber/godog v0.15.1 // indirect
github.com/cucumber/messages/go/v21 v21.0.1 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/deckarep/golang-set/v2 v2.8.0 // indirect
github.com/ethereum/c-kzg-4844/v2 v2.1.5 // indirect
Expand All @@ -49,9 +52,13 @@ require (
github.com/gagliardetto/solana-go v1.14.0 // indirect
github.com/gagliardetto/treeout v0.1.4 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/gofrs/uuid v4.3.1+incompatible // indirect
github.com/google/go-configfs-tsm v0.2.2 // indirect
github.com/google/logger v1.1.1 // indirect
github.com/gorilla/websocket v1.4.2 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-memdb v1.3.4 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/holiman/uint256 v1.3.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.18.1 // indirect
Expand All @@ -69,6 +76,7 @@ require (
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/streamingfast/logging v0.0.0-20250918142248-ac5a1e292845 // indirect
github.com/supranational/blst v0.3.16 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
Expand Down
33 changes: 33 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,20 @@ github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 h1:zuQyyAK
github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ=
github.com/consensys/gnark-crypto v0.19.2 h1:qrEAIXq3T4egxqiliFFoNrepkIWVEeIYwt3UL0fvS80=
github.com/consensys/gnark-crypto v0.19.2/go.mod h1:rT23F0XSZqE0mUA0+pRtnL56IbPxs6gp4CeRsBk4XS0=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.7 h1:zbFlGlXEAKlwXpmvle3d8Oe3YnkKIK4xSRTd3sHPnBo=
github.com/cpuguy83/go-md2man/v2 v2.0.7/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/crate-crypto/go-eth-kzg v1.4.0 h1:WzDGjHk4gFg6YzV0rJOAsTK4z3Qkz5jd4RE3DAvPFkg=
github.com/crate-crypto/go-eth-kzg v1.4.0/go.mod h1:J9/u5sWfznSObptgfa92Jq8rTswn6ahQWEuiLHOjCUI=
github.com/crate-crypto/go-ipa v0.0.0-20240724233137-53bbb0ceb27a h1:W8mUrRp6NOVl3J+MYp5kPMoUZPp7aOYHtaua31lwRHg=
github.com/crate-crypto/go-ipa v0.0.0-20240724233137-53bbb0ceb27a/go.mod h1:sTwzHBvIzm2RfVCGNEBZgRyjwK40bVoun3ZnGOCafNM=
github.com/cucumber/gherkin/go/v26 v26.2.0 h1:EgIjePLWiPeslwIWmNQ3XHcypPsWAHoMCz/YEBKP4GI=
github.com/cucumber/gherkin/go/v26 v26.2.0/go.mod h1:t2GAPnB8maCT4lkHL99BDCVNzCh1d7dBhCLt150Nr/0=
github.com/cucumber/godog v0.15.1 h1:rb/6oHDdvVZKS66hrhpjFQFHjthFSrQBCOI1LwshNTI=
github.com/cucumber/godog v0.15.1/go.mod h1:qju+SQDewOljHuq9NSM66s0xEhogx0q30flfxL4WUk8=
github.com/cucumber/messages/go/v21 v21.0.1 h1:wzA0LxwjlWQYZd32VTlAVDTkW6inOFmSM+RuOwHZiMI=
github.com/cucumber/messages/go/v21 v21.0.1/go.mod h1:zheH/2HS9JLVFukdrsPWoPdmUtmYQAQPLk7w5vWsk5s=
github.com/cucumber/messages/go/v22 v22.0.0/go.mod h1:aZipXTKc0JnjCsXrJnuZpWhtay93k7Rn3Dee7iyPJjs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand Down Expand Up @@ -105,6 +113,9 @@ github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78=
github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E=
github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0=
github.com/gofrs/uuid v4.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gofrs/uuid v4.3.1+incompatible h1:0/KbAdpx3UXAx1kEOWHJeOkpbgRFGHVgv+CFIY7dBJI=
github.com/gofrs/uuid v4.3.1+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
Expand Down Expand Up @@ -134,6 +145,16 @@ github.com/graph-gophers/graphql-go v1.3.0 h1:Eb9x/q6MFpCLz7jBCiP/WTxjSDrYLR1QY4
github.com/graph-gophers/graphql-go v1.3.0/go.mod h1:9CQHMSxwO4MprSdzoIEobiHpoLtHm77vfxsvsIN5Vuc=
github.com/hashicorp/go-bexpr v0.1.10 h1:9kuI5PFotCboP3dkDYFr/wi0gg0QVbSNz5oFRpxn4uE=
github.com/hashicorp/go-bexpr v0.1.10/go.mod h1:oxlubA2vC/gFVfX1A6JGp7ls7uCDlfJn732ehYYg+g0=
github.com/hashicorp/go-immutable-radix v1.3.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
github.com/hashicorp/go-immutable-radix v1.3.1 h1:DKHmCUm2hRBK510BaiZlwvpD40f8bJFeZnpfm2KLowc=
github.com/hashicorp/go-immutable-radix v1.3.1/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
github.com/hashicorp/go-memdb v1.3.4 h1:XSL3NR682X/cVk2IeV0d70N4DZ9ljI885xAEU8IoK3c=
github.com/hashicorp/go-memdb v1.3.4/go.mod h1:uBTr1oQbtuMgd1SSGoR8YV27eT3sBHbYiNm53bMpgSg=
github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
github.com/hf/nitrite v0.0.0-20241225144000-c2d5d3c4f303 h1:XBSq4rXFUgD8ic6Mr7dBwJN/47yg87XpZQhiknfr4Cg=
github.com/hf/nitrite v0.0.0-20241225144000-c2d5d3c4f303/go.mod h1:ycRhVmo6wegyEl6WN+zXOHUTJvB0J2tiuH88q/McTK8=
github.com/hf/nsm v0.0.0-20220930140112-cd181bd646b9 h1:pU32bJGmZwF4WXb9Yaz0T8vHDtIPVxqDOdmYdwTQPqw=
Expand All @@ -146,6 +167,7 @@ github.com/holiman/uint256 v1.3.2 h1:a9EgMPSC1AAaj1SZL5zIQD3WbwTuHrMGOerLjGmM/TA
github.com/holiman/uint256 v1.3.2/go.mod h1:EOMSn4q6Nyt9P6efbI3bueV4e1b3dGlUCXeiRV4ng7E=
github.com/huin/goupnp v1.3.0 h1:UvLUlWDNpoUdYzb2TCn+MuTWtcjXKSza2n6CBdQ0xXc=
github.com/huin/goupnp v1.3.0/go.mod h1:gnGPsThkYa7bFi/KWmEysQRf48l2dvR5bxr2OFckNX8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/influxdata/influxdb-client-go/v2 v2.4.0 h1:HGBfZYStlx3Kqvsv1h2pJixbCl/jhnFtxpKFAv9Tu5k=
github.com/influxdata/influxdb-client-go/v2 v2.4.0/go.mod h1:vLNHdxTJkIf2mSLvGrpj8TCcISApPoXkaxP8g9uRlW8=
github.com/influxdata/influxdb1-client v0.0.0-20220302092344-a9ab5670611c h1:qSHzRbhzK8RdXOsAdfDgO49TtqC1oZ+acxPrkfTxcCs=
Expand Down Expand Up @@ -250,12 +272,23 @@ github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible h1
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shopspring/decimal v1.3.1 h1:2Usl1nmF/WZucqkFZhnfFYxxxu8LG21F6nPQBE5gKV8=
github.com/shopspring/decimal v1.3.1/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.7/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/streamingfast/logging v0.0.0-20230608130331-f22c91403091/go.mod h1:VlduQ80JcGJSargkRU4Sg9Xo63wZD/l8A5NC/Uo1/uU=
github.com/streamingfast/logging v0.0.0-20250918142248-ac5a1e292845 h1:VMA0pZ3MI8BErRA3kh8dKJThP5d0Xh5vZVk5yFIgH/A=
github.com/streamingfast/logging v0.0.0-20250918142248-ac5a1e292845/go.mod h1:BtDq81Tyc7H8up5aXNi/I95nPmG3C0PLEqGWY/iWQ2E=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/supranational/blst v0.3.16 h1:bTDadT+3fK497EvLdWRQEjiGnUtzJ7jjIUMF0jqwYhE=
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -148,3 +148,38 @@ subjects:
- kind: ServiceAccount
name: openclaw
namespace: openclaw-obol-agent

---
#------------------------------------------------------------------------------
# Role - LiteLLM secrets read access (scoped to llm namespace)
# The agent reads the LiteLLM master key to write upstreamAuth in pricing
# routes. Scoped to a single secret to prevent broad secret access.
#------------------------------------------------------------------------------
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: openclaw-litellm-secrets
namespace: llm
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["litellm-secrets"]
verbs: ["get"]

---
#------------------------------------------------------------------------------
# RoleBinding - Binds LiteLLM secrets access to agent ServiceAccount
#------------------------------------------------------------------------------
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: openclaw-litellm-secrets-binding
namespace: llm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: openclaw-litellm-secrets
subjects:
- kind: ServiceAccount
name: openclaw
namespace: openclaw-obol-agent
Loading