{"id":4179,"date":"2026-11-08T22:24:28","date_gmt":"2026-11-08T22:24:28","guid":{"rendered":"https:\/\/www.smharter.com\/blog\/?p=4179"},"modified":"2026-11-10T11:04:12","modified_gmt":"2026-11-10T11:04:12","slug":"how-i-added-mutation-testing-to-my-loop","status":"publish","type":"post","link":"http:\/\/www.smharter.com\/blog\/2026\/11\/08\/how-i-added-mutation-testing-to-my-loop\/","title":{"rendered":"How I added mutation testing to my loop"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">WHY<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Mutation testing improves the quality of agentic code generation<\/strong>. Like other deterministic computational verification, its output feeds the agent&#8217;s inferential, self-correcting loops, making the agent &#8220;smarter&#8221;.<br><br><strong>Mutation testing tests the tests, preventing false negatives and ensuring the actual coverage of all execution paths and edge cases<\/strong>. While both goals are traditionally achieved via Red-Green-Refactor micro-testing cycles in TDD and A-TDD, pairing a Test-after approach with mutation testing proved equally effective.<br><br><span style=\"text-decoration: underline;\">Specifically about TDD<\/span><br>Direct experience, experimentation, and peer discussions made me realise that agentic TDD at the unit-testing level burns excessive time and tokens without producing a superior result, compared to letting the agent do Test-after (*). <br><br><span style=\"text-decoration: underline;\">Specifically about A-TDD<\/span><br>As <strong>Uberto Barbini<\/strong>, author of &#8216;<em>Process over Magic: Beyond Vibe Coding<\/em>&#8216; and &#8216;<em>Functions over <\/em>Objects&#8217;, noted, having agents generate a few initial e2e acceptance tests (A-TDD), Test-first alongside the plan helps him validate the agents&#8217; understanding of the work to be done.<br>For the rest, I have found that using a Test-after approach combined with inferential mutation testing achieves the same quality as Test-first.<br><br>(*) <a href=\"https:\/\/www.smharter.com\/blog\/2026\/05\/18\/ai-coding-test-first-vs-test-after-pros-cons\/\" data-type=\"post\" data-id=\"3606\">AI Coding Test-first Vs Test-after: pros &amp; cons<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">WHAT<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><span style=\"text-decoration: underline;\">Inferential Mutation Testing<\/span><br>I began experimenting with mutation testing by instructing the agent to inject bugs into the code. This verified that my e2e tests&#8217; assertions failed as intended rather than passing as false positives. I instructed the agent to do so during the Execution step of a usual PEV (Plan-Execute-Verify) cycle, targeting the e2e tests just created or modified.<br><br><span style=\"text-decoration: underline;\">Computational Deterministic Mutation Testing<\/span><br>Next, I tested my TypeScript and React frontend unit tests using the mutation testing tool &#8216;<a href=\"https:\/\/stryker-mutator.io\/\" data-type=\"link\" data-id=\"https:\/\/stryker-mutator.io\/\" target=\"_blank\" rel=\"noopener\">Stryker Mutator<\/a> for JavaScript&#8217;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When sharing these initial experiments:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Willem van den Ende<\/strong> (Software Developer, Consultant at Living Software, and Trainer who taught me advanced refactoring in 2008 at the University of Insubria&#8217;s European Summer School on Programming) noted that the initial run didn&#8217;t find many mutants because the agent had already caught several missing tests. He found most mutants after adding a coverage gate (in a pre-push hook that blocks code from being merged or deployed if its test coverage falls below a specific threshold).<\/li>\n\n\n\n<li><strong>Oreszt\u00e9sz Margaritisz<\/strong> (Chief Software Engineer at EPAM Systems) highlighted how this inferential approach brings mutation testing directly into the Execution loop, unlike long-running computational mutation testing.<br><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">THE MISTAKES I MADE AND WHAT I LEARNED<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><span style=\"text-decoration: underline;\">Targeted Mutation Testing Runs<\/span><br><br>Mutation testing is common for unit testing and uncommon for integration or e2e tests that are too slow for mutation testing and for which mutation failures, called survivors, are difficult to diagnose and fix.<br><br>Initially, I instructed the agent to run Stryker Mutator during the Execution step, incrementally, that is targeting only newly created or modified frontend unit tests.<br><br>During the Execution step, the agent began running targeted mutation tests as frequently as fast commands like lint, build, and unit tests. This drastically slowed execution and disrupted my process. From this, I realised I need them to run less often, ideally only one time.<br><br><br><span style=\"text-decoration: underline;\">Working Out The Report<\/span><br><br>From the mutation report, I learned about the different failure categories and the appropriate next action for each.<br><br>The mutation report separates killed, surviving, and compiler-caught mutants, automatically grouping survivors by root cause like No Coverage, Runtime Error, or Timeout (for infinite loops or slow tests). The other survivors required manual inspection to be categorised. Here is my summary:<br><br><strong>Killed Mutants<\/strong><br>These are the code mutations the tool injected that caused the unit tests to fail. No action is required since the tests fail as expected because of the injected defects.<br><br><strong>Survived Mutants<\/strong><br>These are defects injected into the code and missed by unit tests; some require fixes, while others don&#8217;t.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Fix Needed<\/strong>\n<ul class=\"wp-block-list\">\n<li>No Coverage: existing unit tests don&#8217;t even cover the mutated code =&gt; a unit test targeting the mutated code must be added<\/li>\n\n\n\n<li>Runtime Error: existing code lacks handling of the runtime error caused by the mutation =&gt; error handling must be added<\/li>\n\n\n\n<li>Stubborn Mutant: a missing assertion makes the test of the mutated code pass =&gt; an assertion must be added to the test<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>No Fix Needed<\/strong>:<br>These survivors may not require a fix; it can be confirmed only by inspecting the mutated code that survived the test to confirm they are one of these:\n<ul class=\"wp-block-list\">\n<li>Equivalent: is a mutant that does not change the observable behaviour of the program <\/li>\n\n\n\n<li>Unreachable: is a mutant whose code cannot be executed because of the code logic or possible input values<\/li>\n\n\n\n<li>Timeout: the mutant may be causing an infinite loop<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Bug Found<\/strong><br>Mutation testing and its reports don&#8217;t directly identify bugs; they only reveal gaps in the unit tests. But &#8230; More on this later.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">A common approach to handling mutation reports is to set a threshold and intervene manually, only when survivor counts get too high.<br><br>I have experimented with a different approach using AI to evaluate every survivor from the report, and classify it into:<br>&#8211; No fix needed: an equivalent, unreachable or a true infinite loop timeout (not just a slow test)<br>&#8211; Fix needed: a stubborn mutant, uncovered code or a runtime error.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The agent classified the survivors, generated fixes where needed, and also uncovered real bugs exposed by the mutants&#8217; inspection. Here are a few examples of bugs identified:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Silenced failures\n<ul class=\"wp-block-list\">\n<li>When calling APIs (as in AWS Cognito API calls to load custom user properties, or to log out)<\/li>\n\n\n\n<li>When  extracting malformed data from the cache <\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Date-time errors\n<ul class=\"wp-block-list\">\n<li>As incorrect filter boundaries<\/li>\n\n\n\n<li>As timezone management errors<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Asynchronous long-running operations completed out of sequence causing a visualisation errors<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><br><span style=\"text-decoration: underline;\"><span style=\"text-decoration: underline;\">Full Mutation Testing Runs &amp; Persisted Triage Record<\/span><\/span><br><br>Given the test gaps and bugs identified and fixed, I wanted to experiment with running full mutation testing across the entire frontend unit test suite. Ideally, as a standard step in the build and test pipeline.<br><br>The full mutation run took around 2 hours, compared with under 15 minutes for the standard build and test pipeline. On top of that, the massive reports generated a huge volume of survivors that required significant effort to analyse, classify, and fix, even with AI assistance.<br><br>Next, I had the agent analyse and classify survivors, apply fixes, and log the triage results to a persistent document. This cut future full runs down to a quick, targeted check of only what changed since the last triage. The work took a few days.<br><br>I initially ran the task with CodeX\u2019s Terra model, quickly burning through my weekly token budget. Then I switched to Luna, which completed the job using only a fraction of those tokens.<br><br>However, the triage document\u2019s references (connecting each mutant with its analysis and related evidence) were too loose. Subsequent runs couldn&#8217;t map analysis rows back to specific mutants, requiring most of the survivors&#8217; analysis to be repeated. This problem defeated the purpose of the triage document, which was to speed up the following full mutation testing execution, analysis and fixes.<br><br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">WHAT ACTUALLY WORKED<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><span style=\"text-decoration: underline;\">Targeted Mutation Testing Runs<\/span><br><br>The full build and test pipeline in a clean environment is the slowest computational, deterministic feedback loop. As such, it runs last in the Execution step. Stryker targeted mutation testing is the second slowest.<br><br>So I instructed the agent to run Stryker targeted mutation testing, analyse the results, and fix the gaps and bugs identified, just once, after coding and testing are complete, as a final check before the full build and test pipeline.<br><br>This solution proved fast and effective by eliminating redundant Stryker runs on the same code and tests.<br>As noted earlier, I paired this deterministic mutation testing on unit tests with inferential mutation testing on e2e tests included in the code + test generation.<br><br>The final result is unit and e2e mutation testing that runs faster and sooner than full computational, deterministic mutation testing on the build and test pipeline. <strong>It is a textbook example of shifting left.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span style=\"text-decoration: underline;\"><span style=\"text-decoration: underline;\"><span style=\"text-decoration: underline;\">Full Mutation Testing Runs &amp; Persisted Triage Record<\/span><\/span><\/span><br><br>Instead of relying on survivor count thresholds, I&#8217;ve perfected using inferential analysis and fixing for mutation test survivors.<br><br>I instructed the agent to structure the mutation testing triage table to include, for every survivor, a reference to the code file, a change-detection hash code, the mutant type and position, and the analysis result.<br><br>I also instructed the agent to archive past Stryker reports on a shared team server, providing a reference to link the code, mutant, analysis, and original report for full context reconstruction.<br><br>As a result, incremental and full Stryker mutation runs can reuse previous survivor analyses, avoiding redundant time and token-consuming work on previously analysed code and survivors.<br><br>The result is: <strong>fast inferential analysis and fixing of reported mutation test survivors<\/strong>.<br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">WHAT&#8217;S NEXT<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><span style=\"text-decoration: underline;\">More evidences of the benefits<\/span><br><br>Over time, I&#8217;d like to see whether running periodic <span style=\"text-decoration: underline;\">full<\/span> mutation testing on the entire frontend code continues to uncover valuable test gaps and bugs, or if the frequent incremental mutation testing, backed by the existing triage analysis, is enough.<br><br>I want further confirmation that fixing these test gaps yields fewer production bugs and ultimately improves reliability.<br><br>I want to see whether continuous gap and bug fixing driven by mutation testing needlessly bloats the unit test suite, increasing run times and making code changes harder, without delivering tangible reliability gains.<br><br><span style=\"text-decoration: underline;\">Extending the experiment to the backend<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, I&#8217;d like to extend mutation testing to the backend using Stryker.NET on our fast unit and integration tests, seeing what gaps and bugs it uncovers there.<\/p>\n\n\n\n<div style=\"height:100px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<hr class=\"wp-block-separator aligncenter has-alpha-channel-opacity\"\/>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-8f761849 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\"><div class=\"wp-block-image is-resized\">\n<figure class=\"alignright size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"420\" height=\"532\" src=\"https:\/\/www.smharter.com\/blog\/wp-content\/uploads\/ai-develop.png\" alt=\"\" class=\"wp-image-2071\"\/><\/figure>\n<\/div><\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<div class=\"text-ads\">\n\t<h3>Ship Production-grade Software Faster with AI. No AI slop.<\/h3>\n<br>\n\t<h3>Drive Value Faster with AI. No AI bloat.<\/h3>\n\t<p>\n\t<br>\n\tSee how we can help across the full AI adoption lifecycle.\n\t<br>\n\t<br>\n\t<\/p>\n\n\t<div class=\"local-scroll\">\n\n\t\t<a href=\"\/ai-services.html\" target=\"_blank\" class=\"btn elastic-btn-mod btn-mod btn-dark btn-medium btn-round\" onclick=\"ga('send','event','Blog tech-ads','Click AI Services','AI Services');\" rel=\"noopener noreferrer\">\n\t\tAI Services\n\t\t<\/a>\n\t<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Experience report: What I tried, the mistakes I made, what I learned from it, and what actually worked when I added mutation testing to my AI-augmented development loop<\/p>\n","protected":false},"author":2,"featured_media":4210,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-4179","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-assisted-coding"],"_links":{"self":[{"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/posts\/4179","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/comments?post=4179"}],"version-history":[{"count":73,"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/posts\/4179\/revisions"}],"predecessor-version":[{"id":4265,"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/posts\/4179\/revisions\/4265"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/media\/4210"}],"wp:attachment":[{"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/media?parent=4179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/categories?post=4179"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.smharter.com\/blog\/wp-json\/wp\/v2\/tags?post=4179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}