BH3D Logo
The Rise of "Test Theater"

The Rise of "Test Theater": When AI Coders Write Tests That Mean Nothing

By Ben Houston, 2025-03-25

A Theater Production while Bugs Walk Around

In the rush to embrace AI coding assistants, we've stumbled into a dangerous illusion: Test Theater – the practice of generating impressive-looking test suites that validate implementation rather than intention. (Inspired by the term Security Theater because it is analogous just in the domain of tests rather than security.)

The False Promise of AI-Generated Tests

It's a tempting workflow: you write some code, then ask Claude, Copilot, or any other AI assistant to "write tests for this function." Within seconds, you get a beautiful test suite with high coverage, edge cases, and neatly organized test blocks.

Your CI pipeline shows a sea of green. Your coverage report hits 90%+. Management is thrilled.

But there's a critical problem: these tests are fundamentally circular.

The Circularity Problem

Unless you are very clear and careful, AI coding assistants at the moment will look at your implementation and write tests that confirm your code does what it already does. They're essentially saying: "This function returns X when given Y, so I'll write a test to confirm it returns X when given Y."

This is the equivalent of a student writing their own exam after seeing the answers. It's a tautology – true by definition, but not validating anything of value.

Real tests should verify that your code meets its requirements and correctly implements its design – not that it consistently produces the same (potentially incorrect) behavior.

How Good Tests Actually Work

Good tests are specification-first, not implementation-first. They should:

  1. Validate requirements: Tests should check that code behaves according to specifications, not just that it's internally consistent.

  2. Protect against regression: Tests should fail when behavior changes in ways that violate contracts with other components.

  3. Document intent: Tests should serve as executable documentation of what the code is supposed to do.

  4. Challenge assumptions: Tests should check edge cases and unexpected inputs that the implementation might not handle correctly.

This is why Test-Driven Development (TDD) advocates writing tests before code. The tests become the requirements, and the code is written to satisfy them – not the other way around.

Why AI Struggles With Meaningful Tests

The whole problems come from using AI assistants as an after thought to write your tests for you. AI assistants at this point don't understand your requirements, system architecture, or business domain unless you explicitly provide this context. When asked to "write tests," they generally just infer expectations from the implementation itself.

They don't know:

  • What the code is supposed to do (only what it currently does)
  • What contracts exist with other components
  • What business rules must be enforced
  • What edge cases matter to your domain

Breaking Free from Test Theater

So how do we use AI assistants responsibly for testing?

  1. Start with requirements: Provide clear specifications to the AI before asking for tests. "This function should validate email addresses according to RFC 5322, reject null inputs, and limit addresses to 254 characters."

  2. Write critical tests yourself: Use TDD principles for core business logic – write the tests first, then the implementation.

  3. Use AI to expand test coverage: Once you have core tests in place, AI can help identify additional edge cases or generate variations.

  4. Review AI-generated tests critically: Ask yourself, "Is this test validating requirements, or just mirroring implementation?" In my experience more than 50% of the tests I've seen written are just mirroring the implementation.

  5. Consider property-based testing: Tools like fast-check (JavaScript) or QuickCheck can generate thousands of test cases to find edge cases your implementation might miss.

The Real Cost of Test Theater

Test Theater isn't just ineffective – it's actively harmful:

  1. False confidence: Teams believe their code is well-tested when it's not.

  2. Refactoring paralysis: Changing implementation becomes difficult because tests are coupled to current implementation details and there will be hundreds of them.

  3. Maintenance burden: You end up maintaining large test suites that provide minimal value. Of course you can use AI to update your implementation tests, but now you are just burning tokens to maintain the theater production.

  4. Missed bugs: Critical issues slip through because tests only validate what the code already does.

Conclusion

AI coding assistants are powerful tools, but they're not magic. They can't understand your requirements or business domain without explicit guidance.

When it comes to testing, these tools should augment – not replace – thoughtful test design based on requirements and system contracts.

The next time you're tempted to ask an AI to "write tests for this code," remember: a test that only confirms your code does what it already does isn't a test at all – it's just Theater.