The Pokemondev.com Harness Hackathon was an experimental hackathon held on June 6 at vyvhouse, the hacker house operated by Hashed. The starting point was simple. “If we can automate everything through AI agents, could we play games the same way?” Pokemon was a familiar game to work with, but the real subject was not clearing the game. It was testing whether an agent could see the world, make decisions, take action, recover from failure, and pursue long-term goals.
This was not simply a competition for automated-play bots. Participants treated Pokemon as a testing ground for agent harnesses. They tested whether an agent could observe the screen, read RAM state, execute short actions, check the result again, and continue its progress through memory and save points. On the surface, the question was “Can AI play Pokemon?” But the questions underneath were much bigger. “Can an AI agent behave like a real user in a complex environment?” “How far can automation enter the game experience?”
Participants quickly realized that Pokemon is harder than it looks. You cannot find your way using coordinates alone, and without looking at the screen, you cannot understand fences, NPCs, doors, signs, or blocked paths. You also have to tell whether a dialogue box is open, whether you are in the battle menu, and whether you are inside or outside a building. One participant said, “I thought the agent was stupid, but actually we just hadn’t given it eyes.” Another said, “One screenshot is better than a hundred lines of logs.” Numerical state is necessary, but the real context of the game appeared on the screen.
The biggest lesson was that how well the harness was designed mattered more than how smart the model was. At first, participants gave the agent long commands like “go north and enter the shop,” but the entire plan fell apart if it went even one tile off. If it bumped into an NPC or a dialogue box appeared, the meaning of the input changed. In the end, participants switched to executing around two to four short actions and then immediately checking the screenshot and state again. One person said, “The hard part wasn’t long-term reasoning. It was the discipline to walk two tiles and stop.” Another said, “We wanted a smarter prompt, but what we actually needed was a shorter action list.”
What was interesting was that this process came to be understood not simply as automation, but as a new kind of game experience. Instead of pressing the buttons themselves, participants gave the agent directions like a backseat coach, corrected its decisions while watching the screen, and observed its failures. One participant said, “Coaching an agent as it plays the game is more fun than playing the game myself.” Another said, “This isn’t auto-grinding. It feels like going on an adventure together while nagging the AI from the side.” They had become meta-players—not players who controlled the game by hand, but players who set goals, gave strategic direction, and supervised the agent’s actions.
At this point, the Pokemondev.com Harness Hackathon also challenged the conventional idea of bots in games. Until now, bots in games have generally been something to stop. They have symbolized automated farming, cheating, damage to competitive environments, and disruption of game economies. But this experiment showed another possibility. In a controlled environment with clear identity verification, public rules, and limited permissions, an agent could become a kind of gamer. Instead of seeing bots only as intruders that must be blocked, we could design them as authenticated agent players, automated players that others can watch, or auxiliary controllers for human gamers.
Of course, the limitations were just as clear as the possibilities. Agents got lost easily, repeatedly ran into the same wall, walked out of a door and then went back in, and sometimes made the wrong choice in the battle menu. But those failures were what made the event fun and educational. As one person put it, “The funnier the scene, the more clearly you could see the system bug.” Pokemon was a good testing ground because you could see an agent’s misunderstandings with your own eyes. Observation failures, action failures, memory failures, and recovery failures that are hard to see in natural-language benchmarks appeared vividly on the screen.
Participants left these comments.
“Pokemon taught me not about Pokemon, but about observability.”
“We weren’t playing a game in which AI plays the game. We were playing the way AI misunderstands the world.”
“The observation loop was harder than Victory Road, and save habits mattered more than the Elite Four.”
“A good agent is less like a genius and more like a careful traveler. It looks at the map, takes one step, and checks again.”
“Saying that bots must be stopped is only half right. The problem isn’t the existence of bots, but bots with no identity and no rules.”
In the end, the question posed by the Pokemondev.com Harness Hackathon goes deeper than “Can AI beat Pokemon for us?” The event asked, “What kind of experience is playing a game with AI?” Automation in the agent era could become not just a replacement for human actions, but a new interface in which humans set the goal, agents carry it out, and humans supervise them again. The same is true in games. Humans can be players who control the game directly, but they can also be players who train agents, watch them, correct them, and go on adventures with them.
This hackathon at vyvhouse showed a very serious possibility through the cute screens of Pokemon. Can agents automate everything? Can games accept that automation? Are bots always the enemy, or can they become new gamers under authenticated and controlled conditions? The answer is not complete yet, but participants confirmed at least one thing. Watching an agent learn about the world while standing beside it and giving it backseat advice is much more fun than you might think.
