Run Minecraft in a Windows Sandbox for Computer Use Agents
This guide explains how to boot a Windows sandbox, install Minecraft Java Edition, and control it with an AI agent via the cua-driver MCP server. It covers setting up a second network interface for internet access, using a software OpenGL driver (Mesa3D) to meet Minecraft's requirements, and driving the game through a vision-capable LLM. The agent loop uses MCP tools to click, type, and navigate, with tips for handling policy restrictions and waiting for loading screens.
The sandbox GPU is the Microsoft Basic Display Adapter, which offers OpenGL 1.1. Minecraft 1.17 and later need OpenGL 3.2, so the game needs Mesa3D's opengl32.dll (llvmpipe), which implements OpenGL in software.
- orbital-decay
But... why? Computer use is probably the least efficient way to do this. There are agentic mods and plugins that are a lot better. See also SkyrimNet, people have been building agents for gaming and roleplay long before tool use was even a thing, let alone the current agentic craze.
- gitowiec
Smells AI...
- lostmsu
Run ... without graphics acceleration.