World News Sentiment
Sentiment-scored news headlines for ~155 countries, refreshed hourly on an interactive map.
Free-tier news and ML APIs have unforgiving quotas, and the obvious design - fetch on request - lets ordinary user traffic burn the day's budget before lunch.
One writer, many readers. An hourly QStash cron is the only process that spends credits: it reserves them in Redis before fetching, selects the countries whose local time is near 6 am, refunds transient failures, and pivots non-English headlines through Azure Translator so a single RoBERTa model does all the scoring. Every browser request is served from a pre-built aggregate behind Vercel's edge cache.
~155 countries scored hourly inside two free tiers, with a 30-day history each. The scoring path was measured rather than assumed - 63.3% three-class accuracy on 300 labelled NewsMTSC sentences - and the read path cannot spend a credit no matter how much traffic arrives.
An interactive world map coloured by the sentiment of each country's latest news headlines, refreshed hourly inside two free-tier news API quotas.
What it does
- Fetches the day's top headlines for ~155 countries from two news providers, staying strictly inside both free-tier quotas via per-provider credit ledgers in Redis.
- Translates non-English headlines with Azure Translator and scores them with the
cardiffnlp/twitter-roberta-base-sentiment-latestRoBERTa model. - Serves an interactive map with per-country headlines and a 30-day sentiment history - refreshed hourly by a timezone-aware rolling scheduler, so every country is fetched near 6 am local time.
Design decisions
Reserve-before-spend credit ledger
Free-tier quotas are unforgiving and a crashed tick must never leave spend unaccounted. Credits are reserved in Redis before the fetches run, so a mid-tick failure can only over-count, never under-count. Countries whose failure was transient get their credits refunded afterwards, reclaiming the slack the daily limits were sized to provide. Each provider has its own ledger, so one API's usage can never throttle the other.
Three-way persistence taxonomy (scored / retry / terminal)
Not every failed fetch means the same thing. A country that produced a score is stored, freshened and marked done. A country that had headline text but failed scoring, or hit a transient provider blip, is left entirely untouched - not marked done, so the next tick retries it while its previous value keeps serving. A terminal outcome (empty, unsupported, 4xx) is marked done and freshened, so a quiet country waits out its cadence instead of burning budget daily.
Timezone-aware rolling scheduler with staleness backfill
Refreshing ~155 countries at once would blow both quotas, and a country's news is most complete in its own morning. Each tick picks the countries whose local time is ~6 am, then spends any leftover budget on the stalest remaining countries - which is also how prior failures self-heal. Backfill will not pull a country forward before its target hour unless it is already ~30 h overdue, so recovery never erodes the schedule.
Read/write separation + edge caching
The read path (/api/sentiment) never calls a news provider: it serves the pre-built aggregate, so
user traffic cannot spend credits no matter how much of it arrives. Responses carry
s-maxage=300, stale-while-revalidate=3600, so Vercel's edge absorbs the traffic and keeps serving
through a Redis outage. Per-country keys have no TTL, so the map degrades to last-good data rather
than to blank.
Two-provider routing with a translation pivot
GNews ranks by popularity but covers far fewer countries (we route our 68 high-priority ones to it); NewsData.io reaches ~155 but returns by recency. High-priority countries are therefore fetched from GNews and the rest from NewsData, behind one enriched shape. On the model side, only one sentiment model is deployed: every non-English headline is pivoted through Azure Translator into English and scored there, which keeps one calibration to reason about instead of many (previously a separate multilingual sentiment model was used for scoring 9 languages and others were translated to English and scored by the current model, but this meant scores were not comparable across the map.)
Model evaluation
The production scoring path was measured against labelled news sentences rather than assumed to work: 63.3% 3-class accuracy (macro-F1 0.627) on 300 NewsMTSC sentences, and the translation round-trip flips a headline's label on 8–15% of items depending on language (worst: Japanese, 15.2%). The model leans negative and is weakest on the neutral class; a threshold sweep confirms the shipped ±0.1 neutral band is near-optimal.