The hidden cost of AI-written tests

I can’t tell you if AI made us ship faster. I can’t tell you if our code got better or worse. I can show you one thing that is not up for debate: the number of tests in our project exploded — and what we had to do so the test suite did not explode with it.

The one thing I can measure

Everybody has an opinion about AI and productivity. I have almost a year of data, and I still do not have an honest answer. Did we ship faster? Maybe. But the team changed, the product changed and the roadmap changed, all at the same time. Is the code better or worse? Ask three developers, and you get four answers. I will not pretend to measure what I can’t measure.

But there is one number that is not an opinion: how many tests we have.

The chart counts test methods. PHPUnit counts more, because one method with a data provider runs once for each data set: today’s 8,127 methods are 11,782 tests in the PHPUnit report.

For five and a half years, the line is almost flat. In Q3 2025, it starts to bend. In Q2 the suite gained 32 test methods; in Q3, it gained 356. Then, in January 2026, we started to let AI write tests, and the line goes almost vertical: from about 2,000 test methods to more than 8,000 in ten months. In the third quarter of 2026 alone, we added 2,651 test methods — more than in the six years before 2026 together.

And the line has not settled yet. It still goes up faster every month. That is why the speed of the suite matters now, and not some day later.

Many tests are good news…

Let me be clear: I like having many tests. When you change one thing, the tests tell you what else you broke — often in places you did not know were connected. With thousands of test methods, the safety net is wide. You can refactor without fear. You can review a pull request and trust the green check.

…until the suite becomes the problem

I will skip maintenance. Old tests that nobody understands, mocks that lie, assertions that check nothing — that is a different article. Even if every single test were perfect, a big suite brings its own problems. They come from size alone.

Slow tests cost money

The first cost is developer time. When the full suite takes too long on your laptop, you stop running it. You push and let CI do it. And nobody stares at a pipeline for 20 minutes. You pick up something else and forget about it. The pipeline goes green — and nothing happens. The task sits there, done and tested, until you remember it and move it on. The tests took 20 minutes. The task lost much more.

The second cost is real money. CI runners on GitHub or GitLab are billed by the minute — or they are your own machines, which you pay for anyway. Do the math for an average team: 30 developers, 5 pushes a day each, 20 minutes per pipeline. That is 3,000 runner minutes a day, more than 60,000 a month. A slow suite is not only slow. It is a line on an invoice that grows with every new test and every new hire.

Tests that talk to each other

Functional and integration tests share a database. One test leaves data behind, and the next one trips over it. You know the symptoms:

• a test passes alone and fails in the full run;
• the suite passes in one order and fails in another;
• you can’t run the suite twice without rebuilding the test database first.

A flaky test is worse than no test. People learn to press “retry”. After a while, a real failure looks like noise too.

The hotfix

Now imagine this. Production is down. You know the fix — it is one line. You commit it, and your CI needs 15–20 minutes to say it is green. Every one of those minutes is downtime. And most of the tests you are waiting for check code that your one-line fix does not touch.

You could skip the tests “just this once”. That is exactly the moment when you most need them.

The project

A few words about the project, so you know what the numbers are about. It is a backend API: PHP 8.4, Symfony 7.4, Doctrine ORM and MySQL 8.0, tested with PHPUnit 11.5 and paratest. It is a modular monolith with 38 modules — no microservices, no exotic tools.

It is not a giant, but it moves fast. In November 2026, src/ has about 6,400 PHP files and 410,000 lines of code. At the start of 2025, it had 170,000. In the last twelve months, around 15 people worked on it regularly. The tests grew even faster than the code: from 110,000 lines at the start of 2026 to 318,000 today. If both keep the pace of the last three months, the tests will have more lines than the code itself around March 2027.

How it looked for us

For a long time, our developers did not write many tests. The main paths were covered, and not much more. That was the flat part of the chart.

Then a few people joined the team — me among them — who came from teams with no testers. In a team like that, nobody checks your work after you. You own all of your code, so you test all of it. We brought that habit with us, and the line moved a little.

The real turning point came later. The team became convinced that AI writes tests well enough to trust them. From that moment, the number of tests did not grow. It exploded.

At first, it did not hurt. Most of the new tests were unit tests, and unit tests are cheap: thousands of them run in seconds. The full suite was barely slower.

But over time, unit tests were not enough. A unit test with mocks tells you that your class does what you think it does. It does not tell you that the whole thing works. So we started to write functional and integration tests, and later even simple end-to-end tests. Those tests need a database, a kernel, real HTTP requests. Each one costs real time.

And the suite got slow. So slow that people stopped watching CI. They pushed, switched to another task, and forgot. The pipeline went green, and the task just sat there, waiting for someone to move it to the QA team. Sometimes it waited much longer than the tests took to run. Nobody was slow at their work. The handoff was.

Step 1: make every test leave no trace

The first job was not speed. It was trust. A test that fails at random teaches people to ignore red builds.

We fixed it with DAMA (dama/doctrine-test-bundle). Every test runs inside a database transaction, and when the test ends, the transaction is rolled back. Whatever the test wrote is gone, and the next test starts from exactly the same database. The trick is that DAMA keeps one database connection alive for the whole test, even when Symfony reboots the kernel — so the transaction survives too. Your application code does not notice: its own transactions become savepoints inside the test transaction.

That removed all three symptoms at once. Test order does not matter anymore, the suite can run again and again on the same database, and a test behaves the same alone and in the full run. It is not magic — anything outside that one connection, like another process, does not see the data — but for functional and integration tests it is exactly what we needed.

Step 2: run tests in parallel

Next, the obvious move: paratest. Instead of one PHPUnit process, it starts several workers and splits the tests between them. More CPU cores, less waiting — at least in theory.

It helped, but not as much as we hoped. The suite got faster, but not “let’s celebrate” faster. And it brought a new problem: tests started failing again.

The reason was the database. All workers used the same test database and touched the same tables at the same time. Thanks to DAMA, every test runs inside an open transaction, and that transaction holds its locks until the test ends. So one worker waited for another, writes inside those transactions were slow, and from time to time two workers blocked each other for good — a lock timeout or a deadlock, and a red build.

There was progress. It was not the progress we expected.

Step 3: one database per worker

At first we tried to fix the failing tests one by one. It worked for a while, until the next red build. Fighting locks was a game we could not win.

So we stopped fighting and removed the cause. If workers fight over one database, give each worker its own. With six workers, we have six test databases: app_test1, app_test2 and so on, up to app_test6. Same schema, same fixtures, no shared tables — and no locks between workers.

The best part: it is almost free to wire up. Paratest gives every worker a number in the TEST_TOKEN environment variable (1, 2, 3, …). Doctrine has an option that adds a suffix to the database name. Put the two together, and each worker connects to its own database without a single change in the tests:

# config/packages/test/doctrine.yaml
doctrine:
  dbal:
    connections:
      default:
        dbname_suffix: '%env(default::TEST_TOKEN)%'

When you run PHPUnit without paratest, TEST_TOKEN is not set, the suffix is empty, and the tests use the normal test database. Nothing breaks for anyone who runs a single test from the IDE.

Back then, only a few tests failed per run — but we also had far fewer tests. So we checked what one shared database would do today. Out of 11,782 tests, 33 to 42 failed in every run, and never the same number twice. With a database per worker, the number is zero, every time. Nobody has to think about it anymore.

The only thing left is to have those databases ready before the run — each one with the full schema and all fixtures. We will come back to that, because it became our next bottleneck.

Step 4: put the database in RAM

Separate databases fixed the locks, and the tests themselves got faster. But the time moved somewhere else: to the start. Before the first test could run, we now had to prepare N databases instead of one.

Our test database is almost empty — just a minimal set of fixtures. The data was never the problem. The schema was. There are a few hundred tables, and creating a few hundred tables, six times over, took a big part of the whole run. Every CREATE TABLE in InnoDB means new files on disk and an fsync to make sure they are really there.

Our first attempt was to tune MySQL. This is test data. We throw it away after every run, so it does not need production guarantees. We turned off what makes writes safe — and slow:

--innodb-flush-log-at-trx-commit=2
--innodb-doublewrite=0
--skip-log-bin

No flush to disk on every commit, no doublewrite buffer, no binary log. It helped a little. But the files were still on disk, and the disk was still the limit.

So we went one step further and took the disk away. We moved the MySQL data directory to /dev/shm — a filesystem that lives in RAM:

# docker-compose.yml
mysql-test:
  image: mysql:8.0
  shm_size: "1gb"
  command: >
    --datadir=/dev/shm/mysql

This one was a bullseye. It did not only speed up the tests. It also made preparing the databases much faster — the same step that had just become our bottleneck.

A fair warning: a lot has changed in our setup since then — the machines, the Docker setup, the suite itself. When we measured it again for this post, the difference was much smaller. Building six databases from scratch took about 39 s on stock MySQL, 32 s with the flags and 28 s in RAM. The tests themselves ran at the same speed in all three. Your numbers will depend on your disk — so measure before you copy this.

Step 5: build once, copy N times

Even in RAM, most of the time still went to preparation, not to tests. And when we looked at it, the reason was almost funny. For every worker database, we did the same thing: create the schema, run the migrations, load the fixtures. Then the same thing again for the next database. Six times the same work, with the same result.

So we stopped doing it N times. We prepare one base database the slow way — schema, migrations, fixtures. Then we copy it into the worker databases with plain SQL, table by table:

CREATE TABLE app_test1.users LIKE app_test.users;
INSERT INTO app_test1.users SELECT * FROM app_test.users;

No PHP, no Doctrine, no fixture logic — MySQL just copies its own data. The copies run in parallel, one process per worker database.

Step 6: don’t build it at all

The last step came from a simple observation. The test database is a pure function of a few files: entities, migrations and fixtures. If none of them changed, the database will be exactly the same as last time. So why build it again?

We take a hash of the contents of exactly those files. That hash is the name of a SQL dump in the cache: fixtures-<hash>.sql. Then:

• The hash is known (the everyday case: you changed a service, a controller, a test): we skip schema, migrations and fixtures completely, and just import the dump.
• The hash is new (you changed an entity or a fixture): we build the database once, the slow way, and save the dump for next time.

On CI, the dump directory is kept in the pipeline cache, so most pipelines never build the database at all.

The key is the list of files. Hash too little, and you load a stale database. Hash too much, and you rebuild for no reason. For us, it is entities, fixtures and migrations — plus the Messenger config, because it creates its own tables. For your project, it may also include Doctrine config or custom DBAL types. Hash whatever really changes the shape of your database.

Put it together with the RAM disk: one import of a small dump, then a parallel copy into N databases — all in memory. Preparing all the test databases now takes literally seconds.

Things that blew up on the way

The steps above work for almost any Symfony project. These did not — they are specific to ours. But I bet you have your own version of each of them.

Locks on files

In production, our locks live in Redis. In tests, they used the flock store — plain files on disk. With one PHPUnit process, nobody noticed. With six workers on the same machine, workers started to take the same lock files at the same time, and some tests failed for no visible reason.

The fastest fix was the in-memory lock store in the test environment:

when@test:
  services:
    Symfony\Component\Lock\LockFactory:
      arguments:
        $store: '@Symfony\Component\Lock\Store\InMemoryStore'

Each worker has its own memory, so workers cannot block each other anymore. The trade-off is clear: a test can no longer check locking between processes. For us, that was a fair price for a stable suite.

Lesson: look for every resource that is shared outside the database — files, local caches, ports, temp directories. Paratest will find them for you, but it will not tell you which one it was.

Xdebug left on

This one is embarrassing, because it is so simple. Our local containers have Xdebug enabled by default — it is a dev image, and step debugging is useful. But Xdebug slows down every PHP call, even when no debugger is connected.

We measured it on the full suite. With Xdebug on, the tests ran about 25–40% slower. Same code, same machine, one switch.

Debug when you need to debug. For the full suite, turn Xdebug off first (XDEBUG_MODE=off is enough) — and make it easy, so nobody has to remember how.

Things that should be cached

This one is specific to our project, but the pattern is not. For years it was invisible. It showed up only when we started to write a lot of controller tests. A small cost per request does not matter in 50 tests. In thousands of tests, it becomes minutes.

Profile one slow test, and look at what happens on every request. For us, it was the OpenAPI validation. Every request and response is checked against our OpenAPI schema — and in the test environment, the cache for it was turned off in debug mode. So every single controller test loaded and parsed the whole schema from YAML again, from scratch. With thousands of controller tests, that is thousands of times the same big file. One switch to keep the cache on in tests, and the schema is parsed once.

The second find was bigger. Some code asked the router for the full route collection on every request, just to read one option of the current route. Matching a URL to a route is fast, because Symfony compiles and caches it. Getting the full route collection is not: Symfony builds it from scratch, with every route in the application, only to give us one value. We replaced it with a small map, built once and cached. The tests showed us the problem first, but the real win was in production. That is a story for a separate post.

How much did these two caches give? We turned them off again for one run: the tests took 2.7 times longer on one process (104 s → 277 s), and 2.1 times longer on six workers (37 s → 77 s). No database trick on this list gave us that much.

Where we are now

To write this post, we took our current suite — 11,782 tests — and turned the optimizations off again, one by one. Same machine, same code, Xdebug off in every run. DAMA and the in-memory locks stayed on, because without them the results are not stable enough to measure.

From more than five minutes to under one — 6.5 times faster, with every test green. And one more row that is not in the table: one shared database for all workers. It is about as fast as one database per worker, but 33 to 42 tests fail in every run.

A few things we learned on the way:

1. Make tests independent first, fast second. A slow test costs you minutes. A flaky test costs you trust, and a team that does not trust its tests stops looking at them.
2. Measure, do not guess. Some of our numbers surprised us, even after we had done all of it ourselves.
3. Preparing the tests is part of the tests. Building databases took more time than some of the tests. Nobody looks at that part, because it is not in the PHPUnit report.
4. Look at your own code, not only at the infrastructure. Two small caches gave us more than every database trick together. That problem was only visible because we had so many tests.
5. Make the fast way the default. One command that does the right thing is better than a wiki page that nobody reads.

So, back to the beginning. I still can’t tell you if AI made us ship faster. But I can tell you this: we have four times more tests than at the start of the year, and the full suite runs in less than a minute. If AI is going to write a large part of your tests, that is the part you have to take care of yourself.