I Built an AI Companion That Actually Plays With You
I built a low-latency AI companion that plays Skyrim with me
Varkos is a real-time AI gaming companion that plays alongside you in Skyrim, handling complex commands, combat, and even hide-and-seek with low latency and local inference. It uses a hybrid system called ALE (Action Latent Encoder) that combines embeddings, classifiers, and rules to understand natural language and act on the game world. Varkos also evolves his personality based on your interactions, and can exist outside the game in 'void mode.' The system is designed to be pluggable into other games.
When immersion kicks in, it should feel like you are not playing alone.
- usernomdeguerre
>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it)
I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old.
unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?
- Xx_crazy420_xX
Very promising project! The latency is great and looks pretty polished for solo work.
> ALE is designed to be largely invariant to phrasing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it doesn't matter, it will still understand you
> It creates embeddings from the full text as well as its extracted structure
I get how you utilize embeddings, but most interesting part for me is how you decompose multiple commands? You decompose sentences before embedding?
- seabombs
I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game.
The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.
- mike_hearn
This is the sort of thing that the GPT-Live model would excel at when it launches. I'm not sure you'd need the ALE model at all if they launch Live in the API with acceptable pricing as it solves the latency issues with voice entirely. It's an impressive piece of engineering that hasn't been discussed much yet, I guess because it's only a ChatGPT feature and not accessible to developers. The model can dispatch tools asynchronously while speaking which would be ideal for a game.
What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.
- rao-v
I like the approach here. It’s cleverer than just “dump context to LLM”. I couldn’t quite figure out if it genuinely runs purely locally though.
If nothing else - this is how NPCs should work in games moving forward!