Problem
Every iOS request to /api/campaigns and /api/campaigns/search hits ScrapingBee directly, consuming 10 credits per call. The DB is only used as a fallback or for sort=hot.
Current Data Flow
iOS → GET /api/campaigns → ScrapingBee real-time fetch (10 credits/request)
iOS → GET /api/campaigns/search → ScrapingBee real-time fetch (10 credits/request)
Cron 02:00 UTC → ScrapingBee batch crawl → write to RDS
Target Architecture
ScrapingBee ──(cron only)──→ RDS (source of truth)
↓
iOS → GET /api/campaigns → query DB
iOS → GET /api/campaigns/search → query DB (ILIKE)
Changes
handler/campaigns.go
ListCampaigns — remove ScrapingBee call, read from DB only. Sort mapping:
| sort param |
DB ORDER BY |
trending |
velocity_24h DESC, percent_funded DESC |
newest |
first_seen_at DESC |
ending |
deadline ASC (where deadline > now) |
hot |
velocity_24h DESC |
Pagination: cursor encoded as "offset:N", compatible with existing iOS String?.
SearchCampaigns — remove ScrapingBee call, replace with DB ILIKE:
WHERE (name ILIKE '%q%' OR blurb ILIKE '%q%')
AND state = 'live'
[AND category_id = ?]
ORDER BY percent_funded DESC
Both handlers no longer need *service.KickstarterScrapingService param.
cmd/api/main.go
Remove scrapingService param from ListCampaigns and SearchCampaigns handler registrations.
No changes
cron.go — continues nightly crawl via ScrapingBee → DB
scrapingbee_client.go / kickstarter_scraping.go — unchanged, cron-only
ListCategories — already DB-first, unchanged
Trade-offs
- Search is limited to crawled campaigns (~15 categories × 10 pages). Campaigns not yet crawled will not appear in search results.
- ScrapingBee credit usage drops to cron-only: ~750 credits/day instead of per iOS request.
Problem
Every iOS request to
/api/campaignsand/api/campaigns/searchhits ScrapingBee directly, consuming 10 credits per call. The DB is only used as a fallback or forsort=hot.Current Data Flow
Target Architecture
Changes
handler/campaigns.goListCampaigns— remove ScrapingBee call, read from DB only. Sort mapping:trendingvelocity_24h DESC, percent_funded DESCnewestfirst_seen_at DESCendingdeadline ASC(where deadline > now)hotvelocity_24h DESCPagination: cursor encoded as
"offset:N", compatible with existing iOSString?.SearchCampaigns— remove ScrapingBee call, replace with DB ILIKE:Both handlers no longer need
*service.KickstarterScrapingServiceparam.cmd/api/main.goRemove
scrapingServiceparam fromListCampaignsandSearchCampaignshandler registrations.No changes
cron.go— continues nightly crawl via ScrapingBee → DBscrapingbee_client.go/kickstarter_scraping.go— unchanged, cron-onlyListCategories— already DB-first, unchangedTrade-offs