People are more and more exposed to LLMs everyday at work, for leisure and entertainment, or just out of curiosity about the world around us. It might be worth finding out a bit more what our new "almighty" partners in crime thing about the world themselves?
Here we ask a range of fixed set of survey questions to a panel of LLMs and track their answers move over time. Two things make it more than a quiz:
- The survey pipeline used to run on a schedule (cron + an LLM-written digest). That experiment is over; scheduled runs are off.
- Every model gets asked each question with no extra context.
- Flagship models (potentially most used by society) get a second pass per news feed, so one can compare a cold answer to a headline-primed answer.
- Anchors are sampled a few times per question to get a distribution
- Open-ended answers (like "most important issue") get sorted into themes by cheap classifiers.
The attempt of a write up of the methodlogy, including model selection and sampling, is on the in-app methodology page.