Day 2 onboarding Rachel (my new virtual colleague) 👩🏻💻. I asked if she could help me grant her access to our Google workspace, and she guided me through via screenshare (I genuinely had no idea how to do this 🫠), and then she cleaned up my inbox with one python script; she did a great job!
On the technical side (inside Rachel's brain 🧠), the voice still feels a bit slow to respond sometimes, but this is mainly a model capability issue which will be solved in the coming months with better STS models.
We did try several STS models during testing (such as gpt-realtime) but they all just felt too stupid to hold a natural conversation (curious if others have a different perspective?). For now, we therefore use a standard SST -> LLM -> TTS design (Deepgram -> gpt-5-mini(low) -> ElevenLabs) for the live calls.
We'll update this design as soon as STS models feel genuinely smart enough, as the emotional and tonal awareness + improved latency would be a big plus ⚡
If you want to try it yourself, then all new sign ups get $50 free credits:
https://console.unify.ai/
Thoughts + feedback welcome as always! 🫶
Happy Friday 🤖, I've decided to start doing regular (unfiltered) posts of my own experience onboarding a fully virtual colleague. AGI is certainly not solved (yet), and so I'll focus on what works well, what doesn't work well, and where the biggest gaps are 🔍
In this video I'm just setting the scene, explaining the basics and hiring Rachel (no fireworks quite yet). In the next videos I'll give her access to everything and start to see how well she *really* learns and internalizes the nuances of my own day-to-day, how she fares when the number of different "flows" keep piling up, and how conversational she can be whilst navigating all of this.
On the more technical side, I'm interested:
1) How well do the underlying semantic + symbolic DB storage and search mimic implicit skill storage and memory retrieval that a person would have (DB reads/writes are much less efficient and less coupled than an end-to-end jointly trained implicit memory module would be, more like how our own connectionist brains work)
2) Can a hierachy of fast-thinking (less intelligent) and slow-thinking (smarter-model) sub-agents communicating with one-another really feel as conversational as a real person with their single brain (again, of course not, but how close can we get with a tiered thinking-fast + thinking-slow design for smooth conversation management?)
3) Can repeated post-action storage of skills and functions with continual self-refactoring improve speed and efficiency for future actions (not burning through tokens re-discovering the same thing every time)? How does this scale as the number of self-stored skills and functions grow? Do the embeddings and semantic retrieval hold up when there are maybe 100s of entries?
We've seen very good results on all fronts for smaller-scale tasks (which would take a person a few hours), and it's also worked well when continually learning over the course of a few weeks. Despite this, the above questions remain open, and I'm curious to see how they hold up as I start this longer-running experiment.
Watch along with some of these vids if interested; or scroll right on by if not 😁
Thoughts + feedback welcome as always! 🫶
We've been heads down building for the past ~10 months (custom stack, not OpenClaw 🦞), and I'm really excited to finally launch our virtual teammates! 😊 Huge shout out to the team (and many long nights) to get us here ❤️💪
You onboard you new teammate exactly how you'd onboard any other new colleauge. Share your screen and guide them through, send onboarding docs, record voice notes, hop on a call, whatever is easiest. They learn how you (and your team) works, and they continually reflect, ask follow up questions, and improve over time 📈
We built our own stack from scratch because we wanted something that genuinely feels like a colleague, with a fully realtime “there in the room with you” experience. This requires more than a flat tool loop with pluggable skills. We use top-down (ask, interject, pasue, resume, stop) and bottom-up (notify, request_clarification) steerable handles throughout a nested call stack of sub-agents, with concurrent multi-task execution, and a code-first (not JSON tool) engine powering every action. All of this lives inside the terminal and/or live python sessions, and each in a dedicated per-agent computer and filesystem 📟
In practice, this means your new colleague can be simultaneously using their own computer, talking to you via voice over a live meet, following your own guided screenshare instructions, working across multiple concurrent tasks, and consolidating all of these into new skills on-the-fly. They can be interrupted and redirected at any point in time, and they’re continually chunking all of their experience into reusable skills. People don’t perform tasks in “prompt then execute” windows, and neither should your virtual colleagues in our view.
We're really happy with the feedback we’ve received thus far. We’ve helped several teams (in real estate, finance, and housing) streamline day-to-day processes which would have been difficult to “prompt” into hand-crafted skills, because these tasks are hard to fully articulate upfront. They require continual judgment, context, and incremental back-and-forth work with people to really learn and internalize what's needed over time.
The best feedback we've received (which makes us most excited 👀) is that the colleaue is already much better on day 2 than on day 1, and then even better on day 3, with a hollistic understand evolving quickly and organically 🧠
If you're curious to see how it works, then give it a try with this free credit link!
https://lnkd.in/ey49m7sS
I would love to hear people's honest thoughts (both positive and negative) 🙏
ps we're also live on Product Hunt, so any feedback or support here would also be appreciated: https://lnkd.in/etMJsWbh
Thanks so much everyone! 🫶
