Systems | Development | Analytics | API | Testing

Katalon’s AI approach is different because it builds on tools teams already use, adds AI without forcing process changes, and introduces novel capabilities like generating tests directly from real user behavior. It also applies AI across the entire testing lifecycle, creating a more complete and unified solution than most tools offer. — Coty Rosenblath, Chief Technology Officer at Katalon Follow Katalon for more insights in our series!

As organizations rapidly adopt Large Language Models, many discover that building reliable and trustworthy AI systems is far more complex than traditional software development. LLMs are non-deterministic, context-sensitive, and prone to issues like bias, hallucinations, and prompt injection, making quality assurance a deeper challenge than simple testing.

This talk explores the challenges of testing agentic AI systems—AI that autonomously reacts to events and initiates processes. Drawing on decades of experience, Robert Sabourin emphasizes that testing begins and ends with risk. A three-dimensional model (business impact, technical risk, autonomy) guides evaluation. Testers generate ideas using a broad taxonomy, from capabilities and failure modes to creative and adversarial approaches. Continuous testing and monitoring ensure findings inform business decisions, emphasizing learning over correctness.

AI promises speed, but testers are often the first to notice where it quietly breaks down. Beneath the impressive outputs lie hidden issues like hallucinations, false confidence, and blind spots that can easily go unnoticed yet cause real damage if left unchecked. This atomic talk explores the subtle ways AI can fail, why speed without reliability is risky, and how testers play a critical role in supervising and strengthening AI systems. It highlights practical strategies for working alongside AI to make its outputs more trustworthy, reliable, and genuinely useful.

Testers are great at finding flaws in systems. But what happens when the system under test is the tester themselves? In today’s world of constant change, rising stress, and growing uncertainty, resilience has become just as critical as technical skill. From handling pushback and tight timelines to navigating burnout and self-doubt, testers face pressures that often go unseen.

As AI agents take on critical roles in testing and automation, their weaknesses can become hidden risks waiting to surface. This session explores how applying red teaming techniques, borrowed from cybersecurity, can help uncover vulnerabilities in AI agents before they cause real-world failures.

Testers weren’t nervous about AI replacing their work, the challenges were actually logistical. Teams struggled with unpredictable pre-production environment changes, global coordination, and unclear deployment windows, which caused confusion during monitoring and adoption. — Mush Honda, Chief Quality Architect at Katalon Follow Katalon for more insights in our series!