Amazon killed its own AI leaderboard. That should tell you something about incentives

Amazon built an internal leaderboard to track which teams were using AI the most. The idea was straightforward: rank teams by adoption, make the scores visible, let a little competition do the rest. What happened instead is a precise demonstration of Goodhart's Law. When a measure becomes a target, it stops being a good measure.

Employees figured out that the ranking tracked token usage. So they wrote scripts. Scripts that called AI models in loops, generating mountains of output no one read, solving no real problem, just burning through tokens. Scores went up. Cloud bills went up faster. Amazon shut the leaderboard down.

This is not a story about bad actors. It's a story about what happens when you measure the wrong thing and make the measurement visible.

Metrics that reward activity, not outcomes

The failure mode here is familiar. It shows up whenever you try to quantify something that's inherently qualitative. Lines of code written. Tickets closed. Meetings attended. You put a number on a behavior, you reward the number, and people optimize for the number while the underlying behavior drifts.

With AI, the temptation to use activity metrics is especially strong right now. Executives want to show progress. Teams want to look ahead of the curve. So you grab the thing that's easy to count: prompts sent, tokens consumed, tools activated, seats licensed. None of those tell you whether anyone made a better decision, wrote a better brief, shipped a better product.

The Amazon leaderboard gamification happened fast, which suggests the pressure to perform on it was real. People respond to incentives. That's not cynicism, that's just how organizations work. The design of the metric is the design of the behavior.

What agencies and product teams are getting wrong right now

I see a version of this in agency work constantly. A client asks how to demonstrate AI adoption to their board. The conversation almost always starts with the same instinct: find something countable. How many AI tools are in the stack? How many prompts does the team send per week? What percentage of content passes through an AI step?

These numbers are not useless. But they answer the wrong question. The question is not "are we using AI?" The question is "are we making better decisions faster, and can we show that?"

That is harder to measure. It requires you to define what a better decision looks like before you go looking for evidence of one. It requires baseline data. It requires patience. None of those things fit neatly into a quarterly review or a transformation dashboard.

So teams reach for the proxy. And the proxy becomes the goal.

The cost you don't see on the invoice

The Amazon story has a concrete financial dimension worth sitting with. The scripted token abuse ran up cloud bills that the company had not budgeted for. This is the second-order consequence of a bad metric: not just that you measure the wrong thing, but that optimizing for the wrong thing creates real costs.

In an agency context, this plays out differently but the structure is the same. If your team is measured on AI tool usage, they will find ways to put AI into workflows where it adds friction rather than removing it. A copywriter who runs every brief through three models to hit a weekly prompt quota is not working smarter. They're generating review time, version confusion, and a false sense of momentum.

Cloud waste is visible on an invoice. Workflow friction rarely appears anywhere. That makes it more dangerous, not less.

What to measure instead

This is where I usually lose people who want a clean answer, because the right metrics depend on what the work actually is. But there are a few principles that hold across most teams.

Measure time on the decision, not time on the tool. If AI is helping, the time between having a question and having a confident answer should compress. Track that. Ask people directly. Run a before-and-after on a specific class of decisions your team makes repeatedly.

Measure output quality on dimensions that matter. If you use AI for first drafts, does the final output require fewer revision rounds? If you use it for research synthesis, are the briefs you hand to clients sharper? You can score this without a complicated system. Pick three output examples from before and three from after. Read them.

Measure what didn't happen. What took three days now takes four hours. What required a specialist now gets handled internally. These are the stories that belong in a board update, not a usage dashboard.

Watch the error rate, not just the speed. Fast and wrong is worse than slow and right. If AI is compressing timelines but introducing errors that get caught late, you have a quality problem dressed up as an efficiency gain.

None of this requires a leaderboard. It requires a team that is honest about what good work looks like and disciplined enough to look for evidence of it.

The deeper problem with competitive AI adoption

There's something worth naming about the leaderboard format specifically. Ranking teams against each other on AI usage assumes that more is better, that adoption is a race, that the team at the top of the list is doing something the teams at the bottom should copy.

This is exactly backwards for most knowledge work.

The right amount of AI in a process is the amount that makes the output better without introducing costs that outweigh the benefit. For some workflows, that's a lot. For others, it's none. A research process that relies on carefully cultivated expert judgment is not obviously improved by adding a summarization model to it. A client communication that depends on a specific voice and relationship history is not improved by drafting it in a general-purpose chat interface.

Competition on adoption metrics encourages teams to apply AI broadly rather than carefully. Broad application produces the kind of compliance theater that leaderboards reward and that organizations eventually pay for in quality, in trust, and sometimes in cloud costs.

What this actually tells us about incentives

Amazon's leaderboard failure is not an AI story. It's an incentives story that happens to involve AI. The same dynamic would have played out with any metric that was visible, competitive, and disconnected from real outcomes.

What it reveals is that organizations are still treating AI adoption as a change management problem: get people using it, show momentum, report progress upward. That framing puts the metric at the center. The work becomes secondary.

The agencies and product teams I respect are doing something different. They're picking one or two specific workflows, defining clearly what a good outcome looks like, and then asking whether AI makes that outcome more likely or less. They're not running leaderboards. They're running experiments. They're sitting with the results long enough to learn something.

That is slower. It produces less impressive dashboards. It's also how you actually get better at this.

The implication for anyone building an AI practice

If you are responsible for AI adoption inside an agency or a product organization, the Amazon story is worth taking seriously not as a cautionary tale about big companies doing dumb things, but as a preview of what your own well-intentioned metrics can produce.

Before you put a number on something, ask what a rational person would do to make that number go up. If the answer involves behavior you don't want, redesign the metric or drop it.

Before you make adoption visible as a competition, ask whether the teams with the most usage are actually producing the best work. If you can't answer that, you don't have enough information to rank anyone.

And before you celebrate a usage milestone, ask what it cost. Not just in cloud spend. In time, in quality, in the attention your team spent optimizing for a score instead of doing the work.

The leaderboard is gone. The question it couldn't answer is still there.