I love building observability pipelines
At @vercel scale completely new problems arise compared to our 1B events/month at Splitbee
→ A single customer can have billions of datapoints week
→ Data streams are at GB/s scale
→ Achieve e2e latency of <5s
→ Durability is key. We
That was a fun one to investigate 😅
At some point, we floated the idea of the root cause being the LLM (we knew the deployment was created by an Agent) having hallucinated the random GitHub repo…
We started trying to prove that to be wrong (falsification is often a great way
A Vercel user reported an issue that sounded extremely scary. An unknown GitHub OSS codebase being deployed to their team.
We, of course, took the report extremely seriously and began an investigation. Security and infra engineering engaged.
Turns out Opus 4.6 *hallucinated a
Earlier today our oncall team performed the failover we exercised last summer under true emergency conditions.
As designed, Vercel platform serving from our 20 regions was never impacted by control-plane issues and API services+dashboard recovered after the failover.
