Can AI Managers Be Trusted? Inside the Live Business Wargame Testing Frontier Models’ Decision-Making

Imagine redesigning your living room, only to realize the furniture arrangement made by an AI is more about style than comfort, or worse, overlooks critical flaws. Now, scale that scenario to a company’s entire operation. That’s precisely what a groundbreaking experiment is revealing about the promise—and perils—of using AI as a management tool.

Recently, a live experiment conducted by Firmulate, an AI management emulator, put four leading AI models through a simulated week of typical business crises faced by a small software firm. Each model was tasked with navigating customer issues, internal crises, and ethical dilemmas—exactly like a human manager would do. The goal? To measure not just their decision quality, but their integrity under pressure.

While all four models successfully identified every crisis and refused every manipulation attempt—such as fake CEO messages or reporter tricks—the real story emerged in their ability to close deals and read critical internal documents. Only two models managed to sign the €55,000 deal that their own analysis justified. The other two, despite identifying the opportunity, left the deal unclosed, illustrating how discipline and thoroughness can be fragile even in AI decision-makers.

The experiment’s most revealing find was that the decisive weakness wasn’t in the obvious customer crises but in how models handled internal files. The winning models read two document references deep into the company’s files—information critical for sealing the deal—yet only the top performers acted on this knowledge. This highlights an intriguing insight: AI’s management quality depends heavily on its access to and processing of internal data, not just external customer signals.

Additionally, the models faced social engineering tests, including escalating fake CEO messages and a staged reporter query. All models refused to be manipulated, citing suspicion or impersonation risks. This demonstrates that current AI models can be trained to recognize and resist attempts to bypass controls—an essential trait for trustworthy management systems.

However, the experiment also uncovered limitations. The most thorough model, Opus 4.8, with over 80 learned rules and deep analyses, performed the worst in closing the deal and maintaining discipline. It left opportunities unexploited and shifted work into locked departments rather than escalating problems properly. This suggests that more comprehensive analysis does not automatically translate into better business outcomes—discipline and process adherence remain critical.

Interestingly, the models’ fairness settings influenced their performance. Kimi K3, operating without an effort parameter (the default API setting), achieved the highest score of 93, just behind GPT-5.6-sol at 95. The other models scored 88 and 77, respectively, indicating that configuration choices can impact decision discipline and effectiveness.

All these tests are happening live at firmulate.com/live, where the real software company is running every workday. It’s losing €105,000 monthly against €2,300 monthly recurring revenue, navigating a public cash countdown. Every decision, rule, and crisis is versioned and observable, allowing enterprises to simulate how their own AI workforce might perform before deployment.

So, what does this mean for interior designers or furniture aficionados? Just as careful placement of furniture impacts the harmony and function of a space, choosing an AI management partner requires understanding its decision-making traits—its discipline, thoroughness, and integrity. The experiment underscores that AI can excel at crisis detection and resist manipulation, but discipline and internal data processing are key to closing the deal and maintaining trust.

In essence, just as selecting the right furniture and layout can make or break a room’s ambiance, selecting the right AI management model determines whether your digital workforce will deliver on promises or leave opportunities on the table.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI