#17: Testing inthe AI era

Overview: “Testing in the AI era” (Tech League Podcast)

This episode explores how software testing must evolve as AI is increasingly used to generate both application code and tests. A recurring theme is that “testing” is really about encoding requirements—i.e., what the system must do—into executable checks, often via unit, integration, or UI automation. It also highlights that in AI workflows, the testing strategy changes depending on how AI produces code (e.g., small iterative chunks vs. long agent chains).

---

Main technological concepts & product/testing implications

1) Why testing becomes more important with AI-generated code

• AI can generate large amounts of code quickly, increasing the chance of defects and architectural issues.
• Testing must scale, but more importantly: AI-generated code still needs human review, especially for maintainability and correct architecture.
• Coverage can’t just mean “more tests”; it must be at the right abstraction level so it remains reviewable and maintainable.

2) Three testing areas discussed

1. Test the code AI generates (unit/integration/UI).
2. Test AI/agents themselves (agents doing the work correctly).
3. Generate tests with AI, along with the risks of producing low-quality or overly low-level tests.

3) “Guide the AI with tests” (and also with architecture)

The speaker argues you should guide AI not only with requirements, but also with tests. However:

• Test quality depends heavily on whether AI is instructed to produce good architecture.
• If AI produces poorly structured code, it often produces tests that are hard to review and maintain.
• Recommended approach: architecture first, then adapt test generation to that architecture—especially when working in “tiny chunks.”

4) Testing approach should match the AI workflow style

A guest contrasts two AI operating styles:

• Long-running chained agents (hours, multiple agents).
• Tiny chunk iteration (short, controlled changes).

Testing strategy changes accordingly:

• In tiny-chunk workflows, tests should focus on what changed and what the next architecture step requires.

---

TDD vs “architecture-first” (and why pure TDD can be awkward)

The discussion questions whether classic “tests first” TDD works well with AI because:

• Without application structure, it’s unclear what unit tests should target.
• AI doesn’t naturally follow the “minimum code to satisfy the test” evolution humans do in IDE-driven tiny steps.

A practical middle ground emerges:

• Use architecture-first (or “requirements encoded early”) rather than forcing AI into strict test-first loops that can create awkward cycles.

---

Unit testing and UI testing: abstraction levels matter

5) UI automation with Playwright: AI can create usable tests if app code is structured well

The guest describes AI-generated Playwright UI tests as:

• Generic, but maintainable because the application code was structured well (e.g., components/pages, routing, and abstractions).

Key point:

• Application code abstractions (page objects/components) strongly influence whether generated UI tests become maintainable.

6) Against “self-healing tests”

There’s strong criticism of self-healing tests because they hide failures. The advocated approach:

• When app behavior or locators change, update tests intentionally alongside code.
• Tests represent requirements; they should fail when requirements and implementation diverge, prompting review rather than silent healing.

7) Why AI-written UI/execution code can become unmaintainable

AI often writes UI/execution code at too low a level (e.g., brittle locators and deep UI traversal). The speaker argues for:

• Abstraction layers (page objects/component abstractions), so changes in IDs/structure don’t require rewriting everything.

---

“Coverage” as information: stability vs exploration

8) Testing is framed as information acquisition

Testing is not only regression prevention—it’s also about discovering gaps/missing behaviors (“new surprises”). AI changes how much regression is needed vs. where effort should be invested.

9) Stable interfaces vs fast-changing parts

A proposed strategy for AI-driven iteration:

• Put strong execution coverage early around stable seams, such as:
  • interfaces
  • API contracts
  • integration points that shouldn’t change often
• Allow less rigid coverage elsewhere to avoid slowing iteration.

Goal:

• A qualitative shift toward fewer, higher-value tests, rather than generating “100 more tests.”

---

Agentic / tool-based testing challenges

10) Determinism and reproducibility

Natural-language or agentic “run tasks” approaches may be non-deterministic (different call order or inputs each run). For repeatability:

• Convert natural language into domain-specific steps (and ultimately code) so execution becomes deterministic and reviewable.

11) “Adversarial AI testing” feasibility

The guest argues purely “adversarial” approaches have limits:

• If the same model generates both code and tests, they may share the same failure modes.
• To explore new holes, you likely need different models/tools/state/history.
• Many vendors try “noise-reduction” scanning + generated tests, but a major unresolved problem is determining coverage completeness (“what space did it explore?”).

---

Security testing: AI can generate exploits and new testing opportunities

12) Security testing changes in surprising ways

AI is highlighted as especially valuable in security testing because it can:

• Generate proof-of-concept exploits for CVE findings, turning scan results into actionable exploit code.
• Move toward “reachable, exploitable assessment” and impact evaluation.

13) Red teaming / penetration testing limits

There’s skepticism that AI can fully replace expert security testing, but it can assist:

• Security scanning tools may be more reliable because they are domain-specific and may use retrieval (RAG) and other mechanisms beyond a base LLM.
• Exploitability should be verified with human/technical oversight:
  • don’t trust findings without evidence (ideally multiple exploitation paths, not a single simplistic answer).

---

Guidance on building AI-assisted test frameworks

14) Use “buzzwords”/architecture principles encoded in agent instructions

You don’t necessarily need special tools—what matters is giving the agent:

• effective testing architecture concepts (e.g., page objects, interfaces—but not “testing page objects” themselves)
• linting, unit tests, etc.

Example:

• Put real patterns and examples into the codebase or agent instructions (e.g., agents.md) so agents follow the project’s conventions.

15) Agentic QE fleet (mentioned)

The guest recommends experimenting with Agentic QE fleet, an open-source repo/collection of skills/agents intended for:

• auditing
• maintaining history over time (history integration is a work in progress)

---

Speakers / sources

• Toby Sears — host (agentic coding practitioner mentioned)
• Christian Fisher — co-host
• Alan Richardson (eviltester.com) — testing specialist/guest (main source for testing/AI/testing-architecture perspectives)