Recap of the Event

“In the beginning, there was man. And for a time, it was good. But humanity's so-called civil societies soon fell victim to vanity and corruption. Then man made the machine in his own likeness. Thus did man become the architect of his own demise” - The Animatrix, 2003.

Today, the global proliferation of AI cannot be stopped. In the blink of an eye, every tech organisation will be using GenAI and Machine Learning as part of their core tech stack. The speed of this adoption can seem intimidating, and as testers, we experience feelings of paranoia and dread at the growth of this technology.

Questions and concerns abound:

• Will it replace me?
• How can I possibly learn it as fast as it’s evolving?
• Are testers going to become AI testers?

How can I possibly continue to do my job, while at the same time learning the difference between linear regression and dimensionality reduction?

These are just some of the questions testers have been asking themselves since the new generation of AI technology was born.

The mission of our meetup was to demystify some of the more common AI practices. We wanted to show that, as daunting as this new paradigm is, we already have the skills to support dev teams in the usage and development of AI-driven applications.

Below is a summary of what we did on the night. It is intended to recap the event for those that attended and provide useful information for both attendees and those that couldn’t make it.

 Activity 1: Riskstorming the JARVIS AI system

Imagine being tasked with building JARVIS, Tony Stark's super-intelligent AI. That's exactly what we did in our first activity! We used our risk assessment skills to identify potential pitfalls and brainstorm testing solutions for a system unlike anything we've encountered before. This activity highlighted the adaptability of testers and the fact that core testing principles apply even to cutting-edge AI.

Learning Materials -

Risk storming:

• https://www.ministryoftesting.com/testsphere/riskstorming

Accessibility was a recurring subject in the groups:

• https://equalentry.com/ai-for-accessibility-opportunities-and-challenges/#:~:text=and%20public%20infrastructure.-,Challenges%20of%20AI%20for%20Accessibility,associations%20resulting%20in%20unpredictable%20outputs

…as was Observability:

• https://newrelic.com/blog/nerdlog/observability-for-all

Security popped up, with people discussing data flow and the risks involved with mass usage of LLMs: 

• https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/how-should-we-assess-security-and-data-minimisation-in-ai/

Staying with security, we got some insights around what pen testing looks like for LLMs:

• https://systemweakness.com/large-language-model-llm-pen-testing-part-i-2ef96acb6763

There were some mentions of performance and how our LLM would cope under stress:

• https://www.youtube.com/watch?v=ahnGLM-RC1Y

Activity 2 : Skynet Assignment

Next, we ventured into the world of AI-powered automation. The "Skynet Assignment" challenged attendees to create test automation scripts using only AI tools. This wasn't just about automation; it was about understanding the limitations and potential of AI in testing.

Learning Materials - 

We had an interesting mix of methodologies, from loading up the prompt with too much context, to breaking up the problem into smaller prompts - 

• https://arxiv.org/abs/2201.11903
• https://medium.com/@JerryCuomo/lets-think-step-by-step-advanced-reasoning-in-business-with-chain-of-thought-prompting-dd5ae8a6008
• https://www.linkedin.com/pulse/inputs-outputs-ai-prompt-engineering-gavin-mcmahon-ma3me/

Some of us stayed away from the markup and tried to coax a complete solution out of the LLM, while others used the elements on the site and asked AI to create structured automation around these elements 

• https://www.youtube.com/watch?v=ozqffT6ZF24

ChatGPT was making up references to functions and capabilities that didn’t exist (shock)

• https://news.ycombinator.com/item?id=33841672 

The AIs we used added helpful comments to code block outputs but it still required a reasonable understanding of what it was producing to use them. 

• https://mitsloan.mit.edu/ideas-made-to-matter/how-generative-ai-can-boost-highly-skilled-workers-productivity 

Possibly to embrace the challenge, some attendees used only the tools AI suggested, such as Selenium or Python, even though they lacked a foundational understanding of these frameworks or languages but did have a grasp of another programming language. This may have been to work with the AI rather than have the AI work with them, but it led to a lot more confusion and slowed down progress. 

• https://scienceexchange.caltech.edu/topics/artificial-intelligence-research/trustworthy-ai 

A Winning Example:

Curious how the winning team tackled the Skynet Assignment? Check out their approach:

Chat Log: https://chat.openai.com/chat

Test Script: https://developers.google.com/apps-script/add-ons/how-tos/testing-editor-addons

Some Key Takeaways:

Testers are Essential

• Augmentation, Not Replacement: AI is transforming the landscape of software testing, but it's not a substitute for us. Instead, AI tools are designed to augment testers' capabilities. Let the Herbie Hoover handle the repetitive regression testing and data analysis stuff whilst we get stuck in with the fun stuff.

• Skill Evolution: The integration of AI in testing encourages testers to develop new skills, such as understanding AI algorithms, interpreting AI-generated results, and maintaining AI-driven test systems. Evolve or be extinct is the old adage and it's no more apparent than with AI on the horizon! - Make yourself indispensable! 

• Risk Assessment: Traditional risk assessment techniques remain crucial in the AI testing paradigm. we must identify potential risks associated with AI applications, such as unexpected behaviour or security vulnerabilities, and ensure that just because there's a new tech on block our standards for quality don't slip  

• Critical Thinking: Critical thinking skills are essential for analysing AI outputs, understanding anomalies, and making informed decisions about the system's reliability and robustness.  Who better and more qualified to check if AI tests are accurate than those that have been in the business? It's our job to scrutinize AI behaviour and question at every opportunity

• AI Limitations: AI systems, particularly those based on machine learning, can exhibit unpredictable behaviour and are often considered "black boxes" due to their complex and opaque decision-making processes. We must recognise these limitations and approach testing with strategies that can effectively evaluate the systems

• Bias and Fairness: AI models can inherit biases from training data, leading to unfair or discriminatory outcomes. Testers need to be vigilant in identifying and addressing these biases, ensuring that the AI system provides fair and unbiased results across different user groups and scenarios.

Join the Conversation!

This meetup is just the beginning of the conversation. As AI continues to evolve, testers will be at the forefront of ensuring its responsible and effective use. We have a number of other resources available for people looking to further dive into the topic of Artificial Intelligence from How AI in Software Development Can Transform Your Career to the Core Roles Needed for an RPA Squad. Checkout our resources tab for all further info!