The challenge
Parcel's shopping assistant answers product questions on every page. The year before, Black Friday traffic hit provider rate limits within an hour and the assistant was switched off.
The change
Parcel routed the assistant through Lattis with semantic caching and fallbacks across three providers. Spend limits per key kept the peak inside budget.
router:
cache: semantic
fallback: [open-70b, frontier-xl, fast-8b]
spend_limit: 2500/day
We did not add capacity. We added Lattis.
Hannah Walsh
,
Engineering Manager
,
Parcel
The result
The assistant served 9x normal traffic at 41 ms median latency with no incidents. The cache absorbed more than half of repeat questions at no cost.
