Problem
All cron crawls since 2026-02-27 are returning 0 campaigns and failing the sanity check:
Cron: crawl done, upserted 0 campaigns
ERROR: crawl sanity check FAILED — only 0 distinct campaigns seen (expected >=50).
Possible HTML structure change or ScrapingBee degradation.
Check kickstarter_parser.go [data-project] selector.
The DB currently only has 12 live campaigns (leftover from a prior successful crawl). No new data is being ingested.
Root Cause Hypothesis
kickstarter_parser.go relies on the [data-project] HTML attribute selector to find campaign cards on the Kickstarter discover page. Kickstarter may have changed their page structure, or ScrapingBee is returning a challenge/blocked page with no project cards.
Steps to Debug Locally
-
Fetch a sample discover page HTML via ScrapingBee:
cd backend && go test ./internal/service/... -run TestScrapingBee -v -tags integration
Or write a quick script calling ScrapingBeeClient.FetchHTML() for a discover URL and dump the raw HTML.
-
Inspect the response: does it contain data-project attributes, or has Kickstarter migrated to a different structure (e.g., JSON-LD, a different data attribute, or client-side only rendering)?
-
If structure changed, update the selector/parser in internal/service/kickstarter_parser.go.
-
If ScrapingBee is returning a JS challenge page, try escalating to premium_proxy=true or re-enable render_js=true (at higher credit cost).
Files to Check
backend/internal/service/kickstarter_parser.go — parseDiscoverPageHTML(), [data-project] selector
backend/internal/service/kickstarter_scraping.go — crawl logic and sanity check
backend/internal/service/scrapingbee_client.go — HTTP client, proxy settings
Problem
All cron crawls since 2026-02-27 are returning 0 campaigns and failing the sanity check:
The DB currently only has 12 live campaigns (leftover from a prior successful crawl). No new data is being ingested.
Root Cause Hypothesis
kickstarter_parser.gorelies on the[data-project]HTML attribute selector to find campaign cards on the Kickstarter discover page. Kickstarter may have changed their page structure, or ScrapingBee is returning a challenge/blocked page with no project cards.Steps to Debug Locally
Fetch a sample discover page HTML via ScrapingBee:
Or write a quick script calling
ScrapingBeeClient.FetchHTML()for a discover URL and dump the raw HTML.Inspect the response: does it contain
data-projectattributes, or has Kickstarter migrated to a different structure (e.g., JSON-LD, a different data attribute, or client-side only rendering)?If structure changed, update the selector/parser in
internal/service/kickstarter_parser.go.If ScrapingBee is returning a JS challenge page, try escalating to
premium_proxy=trueor re-enablerender_js=true(at higher credit cost).Files to Check
backend/internal/service/kickstarter_parser.go—parseDiscoverPageHTML(),[data-project]selectorbackend/internal/service/kickstarter_scraping.go— crawl logic and sanity checkbackend/internal/service/scrapingbee_client.go— HTTP client, proxy settings