perf: optimize ScrapingBee credit usage — render_js=false + session_id + ai_selector + usage monitoring

[Jing-yilin]

Background

After thoroughly reading the ScrapingBee docs, several optimizations are available that reduce credit consumption and improve reliability.

Changes

1. render_js=false for HTML cron crawl (5 → 1 credit, 80% saving)

Kickstarter's /discover/advanced page is server-side rendered — the data-project JSON attributes are embedded directly in the HTML before any JS executes. We do not need a headless browser.

	Before	After
Credits/page	5	1
26 categories/month	78,000	15,600
124 categories/month (full)	312,750	62,550 ← fits $49 plan

2. render_js=false for AI search (10 → 6 credits, 40% saving)

AI extraction (ai_query) adds +5 credits on top of the base request. Since Kickstarter pages are SSR, we can use render_js=false (1 credit) + AI (+5) = 6 credits instead of 10.

3. ai_selector to speed up AI extraction

Use ai_selector="[data-project]" to focus AI only on the project cards, reducing token processing time and improving reliability.

4. session_id for paginated crawls

Route all page requests for the same category through the same IP (session valid 5 min). Looks more natural, reduces chance of Kickstarter rate-limiting between pages.

session_id = random int 0–10,000,000, shared across all pages of one category

5. Credit usage monitoring via /api/v1/usage

Log remaining credits at startup and after each crawl. Alert if usage > 80% of monthly limit.

GET https://app.scrapingbee.com/api/v1/usage?api_key=KEY
→ { max_api_credit, used_api_credit, max_concurrency, renewal_subscription_date }

Acceptance Criteria

• FetchHTML uses render_js=false
• ExtractWithAI uses render_js=false + ai_selector
• Cron crawl passes session_id per category
• Startup logs ScrapingBee credit usage
• All tests pass