Generative AI for QA Engineers

Generative AI has moved from experimental chatbots into everyday software, testing and business workflows. Large language models can write, summarise, classify, generate code, work with images and interact with external tools. They are also probabilistic systems that can be wrong, incomplete or overconfident. For QA engineers, the useful mindset is neither “AI can do everything” nor “AI cannot be trusted”, but understanding which tasks fit the technology, how to build reliable AI-assisted workflows and where human verification remains essential.

What is Generative AI?

Generative AI refers to models that can create new content from instructions and context. Depending on the model, the output can include text, code, images, audio, video or actions performed through external tools.

A large language model (LLM) learns statistical patterns in language and generates outputs from the context it receives. Modern instruction-tuned models are trained further so they respond usefully to requests rather than simply continuing text.

Generative AI as a general-purpose technology

Activity	Example
Creation	Draft an email, report, test case or script.
Transformation	Rewrite, translate, simplify or change format.
Extraction	Pull entities, requirements or fields from unstructured text.
Classification	Assign a category, sentiment, severity or label.
Summarisation	Reduce a long document, incident log or discussion.
Question answering	Explain supplied material or organisational knowledge.
Pattern generation	Produce output that follows examples.
Reasoning support	Compare options, identify gaps and propose next steps.

Writing, reading and chatting

Writing is one of the easiest high-value applications because the model can produce a first draft quickly and humans can review it cheaply.

• Emails and internal communication.
• Bug reports and release notes.
• Technical documentation.
• Test strategies and plans.
• Meeting and incident summaries.
• User stories and acceptance-criteria drafts.

Iterative workflow

Intent ↓ AI draft ↓ Human review ↓ Add missing context ↓ AI refinement ↓ Final human approval

Grounded summarisation

Summarise the incident report. Return: 1. Confirmed facts 2. Timeline 3. Systems affected 4. Root cause, if explicitly established 5. Open questions 6. Actions already taken Do not infer missing facts. Mark unsupported conclusions as unknown.

What LLMs can do well

• Generate and transform text quickly.
• Follow patterns demonstrated in examples.
• Draft common programming constructs.
• Extract information from supplied context.
• Generate ideas and alternatives.
• Explain code and technical concepts.
• Produce structured outputs when instructed clearly.

What LLMs cannot guarantee

Factual accuracy

A model can produce incorrect information that sounds plausible.

Complete knowledge

Recent, private or niche information may be missing unless supplied through context, retrieval or tools.

Deterministic behaviour

Small context or wording changes can produce different responses.

Correct reasoning every time

Logical, mathematical and procedural mistakes remain possible.

Automatic business understanding

Internal policies, architecture and team conventions must be supplied explicitly.

Accountability

The person or organisation deploying AI remains responsible for how the output is used.

Prompting fundamentals

A prompt is the set of instructions and context given to the model. Strong prompts reduce ambiguity and make results easier to evaluate.

Task: What should the model do? Context: What does it need to know? Constraints: What rules must it follow? Output: What format should it return? Examples: What does a good result look like?

QA example

Task: Generate API test scenarios for user creation. Context: POST /api/users Required fields: email, name, role Allowed roles: user, admin Constraints: - Include positive and negative cases - Include boundaries and authorisation - Do not invent undocumented fields Output: Scenario | Input | Expected status | Expected behaviour

Prompting tips

• Be specific about the objective.
• Provide only relevant context.
• State constraints explicitly.
• Specify the output format.
• Give examples when style or structure matters.
• Ask the model to identify uncertainty.
• Break complex work into steps.
• Iterate instead of expecting a perfect first response.
• Request alternatives when making a decision.
• Ask for source grounding when factual accuracy matters.

Prompts are part of application logic

When an LLM is embedded in software, prompts become part of the product and should be engineered accordingly.

• Version prompts.
• Review prompt changes.
• Test representative inputs.
• Track regressions.
• Separate trusted instructions from untrusted user data.
• Measure output quality.

Image generation

Generative image models can create new images from text and, in many systems, transform existing images.

Useful QA/software applications

• Early UI concepts.
• Placeholder assets.
• Visual test data.
• Accessibility scenarios involving varied imagery.
• Conceptual diagrams.

Limitations

• Generated details may be wrong.
• Text inside generated images may be unreliable.
• Consistency across many images can be difficult.
• Bias and inappropriate representation can appear.
• Production use may require copyright, brand and policy review.

Using generative AI in software applications

Instead of using a chatbot manually, software can call a model through an API and incorporate the result into a larger workflow.

Application ↓ Instructions + context ↓ Model API ↓ Generated result ↓ Validation / business rules ↓ User or downstream system

Examples

• Summarise support tickets.
• Classify bug reports.
• Generate test-data variations.
• Extract structured fields from documents.
• Answer questions over internal documentation.
• Draft code-review feedback.

Simple conceptual example

def classify_bug(description): prompt = f""" Classify this bug into one category: UI, API, Performance, Security, Data, Other. Bug: {description} Return only the category. """ return llm(prompt)

A production application also needs validation, output parsing, error handling, observability, security, rate limits and evaluation.

Lifecycle of a generative AI project

Define task ↓ Collect representative examples ↓ Build prompt / prototype ↓ Evaluate output ↓ Improve prompt or architecture ↓ Add retrieval/tools/fine-tuning if justified ↓ Test safety and edge cases ↓ Deploy ↓ Monitor ↓ Improve

Start with evaluation

Collect normal, difficult and failure examples before optimising the solution. This prevents endless subjective prompting based only on a few demos.

Start simple

Try prompting a capable model before investing in retrieval, fine-tuning or custom model training.

Evaluate generative AI systematically

Dimension	Question
Correctness	Is the answer factually or functionally correct?
Relevance	Does it answer the actual task?
Completeness	Are important required points missing?
Format	Does it follow the required schema?
Safety	Does it violate policy or expose sensitive information?
Grounding	Is the answer supported by supplied sources?
Latency	Is response time acceptable?
Cost	Is usage economically sustainable?

Cost intuition

Cost depends on model choice, input context, generated output, request volume and supporting infrastructure.

Total cost ≈ request volume × average input cost + request volume × average output cost + supporting infrastructure

Ways to control cost

• Use the smallest model that meets requirements.
• Avoid irrelevant context.
• Limit output length where appropriate.
• Cache reusable results.
• Use deterministic code for simple fixed rules.
• Route easy and hard tasks to different models.
• Measure actual usage.

Retrieval-Augmented Generation (RAG)

RAG combines a generative model with retrieved external information. The application searches a knowledge source and supplies relevant content to the model as context.

User question ↓ Search / retrieval ↓ Relevant documents ↓ Prompt + documents ↓ LLM ↓ Grounded answer

Good RAG use cases

• Internal documentation.
• Policies and procedures.
• Product manuals.
• Frequently changing knowledge.
• Large document collections.
• Answers that need source citations.

QA example

Question: What should happen when an order is cancelled between payment and shipment? RAG workflow: 1. Search requirements and API docs. 2. Retrieve relevant cancellation rules. 3. Answer only from those sources. 4. Return citations. 5. Flag conflicts or missing specification.

RAG does not eliminate hallucinations

• The correct document may not be retrieved.
• An outdated document may rank higher.
• The source may itself be wrong.
• The model may ignore part of the context.
• The model may add unsupported information.

Evaluate retrieval quality and answer quality separately.

Fine-tuning

Fine-tuning adapts an existing model using additional training examples so its behaviour better matches a specialised task or style.

Good reasons to consider it

• Consistent domain-specific output style.
• Reliable adherence to repeated task patterns.
• Specialised classification or transformation.
• Reducing long repeated instructions in every prompt.

Problem	Often try first
Model lacks private product knowledge	RAG.
Output format is inconsistent	Prompting/structured output, then fine-tuning if needed.
Need current information	Retrieval or tools.
Need specialised repeated behaviour	Fine-tuning may help.

Pretraining an LLM

Pretraining is the large-scale foundational training stage in which a model learns patterns from enormous datasets. It requires substantial data, compute and machine-learning expertise.

For most organisations, the practical questions are instead:

• Which existing model should we use?
• Can prompting solve the task?
• Do we need RAG?
• Do we need fine-tuning?
• Should the model use tools?

Choosing a model

Dimension	Question
Quality	Does it meet the task's correctness and reasoning requirements?
Latency	How quickly must the answer arrive?
Cost	What usage volume is expected?
Context	How much input must it handle?
Modality	Do you need text, image, audio or video?
Tools	Must it call APIs or external systems?
Deployment	Hosted API, private cloud, on-premises or local?
Privacy	What data may be sent to the provider?
Reliability	What availability and rate limits are required?

Instruction tuning and RLHF

Base language models primarily learn to predict text. Instruction tuning further trains models on examples of instructions and desired responses so they behave more like useful assistants.

Reinforcement Learning from Human Feedback (RLHF) is one family of post-training techniques that uses human preference information to influence model behaviour. The practical takeaway is that an assistant's usefulness and safety are shaped not only by pretraining but also by post-training and alignment methods.

Tool use

A model does not need to solve every problem using only internal model knowledge. It can use tools to retrieve information or perform deterministic actions.

• Search the web.
• Query a database.
• Call a REST API.
• Use a calculator.
• Read a file.
• Execute tests.
• Create a ticket.

User: What failed in the latest Jenkins build? LLM ↓ requests Jenkins data Jenkins API ↓ Build and test results ↓ LLM summarises failures ↓ User

AI agents

An agent is an AI system that can perform a multi-step workflow by choosing actions, using tools, observing results and continuing until it reaches a goal or stopping condition.

Goal ↓ Plan ↓ Choose tool ↓ Execute action ↓ Observe result ↓ Choose next action ↓ Repeat ↓ Result / human approval

QA example

Goal: Investigate a failed UI regression. Agent may: 1. Read the failed test. 2. Inspect screenshot/trace. 3. Read browser logs. 4. Compare recent changes. 5. Rerun the test. 6. Summarise likely cause. 7. Draft a bug report. Human: Reviews evidence and decides next action.

Day-to-day use in QA work

• Rewrite a bug report.
• Summarise a long ticket discussion.
• Generate exploratory test ideas.
• Explain an unfamiliar error.
• Draft SQL or regular expressions.
• Turn a requirement into Gherkin scenarios.
• Compare two API responses.
• Prepare a sprint QA summary.

Choose task ↓ Remove sensitive data ↓ Provide enough context ↓ Generate draft ↓ Verify ↓ Use / edit output

Analyse tasks, not entire jobs

AI's impact is easier to understand by analysing individual tasks rather than asking whether an entire profession can be automated.

QA task	AI potential	Human value remains in
Draft test cases	High assistance.	Risk, prioritisation and product context.
Write automation boilerplate	High assistance.	Architecture and maintainability.
Investigate a failure	Medium-high assistance.	Root-cause judgement and environment knowledge.
Release-quality decision	Support role.	Accountability and business risk.
Exploratory testing	Useful ideation.	Observation, intuition and adaptive investigation.
Stakeholder communication	Drafting support.	Trust, context and accountability.

Redesign workflows instead of only accelerating old ones

Before

Tester reads 50 bug reports ↓ Manually categorises them ↓ Writes summary ↓ Creates dashboard

AI-assisted workflow

Bug reports ↓ LLM extracts structured fields ↓ Deterministic validation ↓ Human reviews uncertain cases ↓ Dashboard generated automatically

The largest benefits often come from changing the workflow, not merely making each old step slightly faster.

Teams that build generative AI software

Automation potential across sectors

Higher potential characteristics

• Digital input already exists.
• Output can be reviewed cheaply.
• Many examples exist.
• Success can be measured.
• Errors are recoverable.

Lower suitability

• Physical work in unpredictable environments.
• High-stakes decisions with weak verification.
• Tasks requiring deep interpersonal trust.
• Processes with little or poor-quality data.
• Responsibilities where accountability cannot be delegated.

Concerns about generative AI

Concern	Risk
Hallucination	Incorrect information can sound convincing.
Bias	Models can reproduce harmful patterns.
Privacy	Users may expose confidential or personal information.
Security	Prompt injection, unsafe tool use and unintended access create new attack surfaces.
Intellectual property	Generated/supplied content may create copyright or licensing questions.
Workforce impact	Automation changes tasks and required skills.
Overreliance	Users may stop checking a system that is usually correct.

Responsible AI

Responsible AI means designing, deploying and using AI with explicit attention to risk, human impact and accountability.

• Define intended use.
• Identify affected users.
• Measure failure modes.
• Protect personal and confidential information.
• Limit permissions.
• Keep meaningful human oversight.
• Provide appropriate transparency.
• Monitor deployed systems.
• Provide escalation and recovery paths.

NIST's Generative AI Profile complements the AI Risk Management Framework with guidance specifically aimed at managing generative-AI risks across the AI lifecycle.

Artificial General Intelligence

Artificial General Intelligence (AGI) usually refers to a hypothetical AI system with broad and flexible intellectual capabilities across many domains. Current generative AI systems can be impressive generalists, but broad capability does not guarantee human-like understanding, reliability or autonomy.

For engineering decisions, evaluate the systems that actually exist rather than basing requirements on speculative future capability.

Generative AI for QA engineers

QA activity	AI assistance
Requirement review	Identify ambiguity, missing cases and conflicting acceptance criteria.
Test design	Generate candidate scenarios, boundaries and abuse cases.
Automation	Draft Playwright, Cypress, API and unit-test code.
Data generation	Create synthetic data and edge-case variations.
Failure triage	Summarise logs, screenshots and stack traces.
Reporting	Draft sprint QA reports and defect summaries.
Documentation	Create setup instructions and explain test code.
Knowledge	Use RAG over project documentation and test repositories.

Example: AI-assisted test design

Requirement: User can upload a profile image. Supported: PNG/JPEG Maximum size: 5 MB. AI proposes scenarios. QA adds context: - authentication required - image displayed publicly - object storage - EXIF metadata stripped - upload rate limit Final test set: Human-reviewed, prioritised and traceable to risk.

Example: RAG for QA knowledge

Sources: - acceptance criteria - API documentation - test strategy - known defects - release notes Question: What regression areas are affected by changing the authentication token lifetime? RAG assistant: - retrieves relevant documents - identifies session dependencies - cites sources - proposes impacted tests QA: verifies and prioritises coverage.

Example: agentic QA workflow

Trigger: Nightly regression finishes. Agent: 1. Fetch failed tests. 2. Group duplicate failures. 3. Read screenshots/traces. 4. Compare known defects. 5. Identify likely product vs test failures. 6. Draft triage report. 7. Link evidence. Human QA: Reviews classification and decides next steps.

Testing AI-powered applications

AI features need both conventional software testing and model-specific evaluation.

Traditional testing

• API contracts.
• Authentication and authorisation.
• Error handling.
• Performance.
• Availability.
• UI and accessibility.

AI-specific testing

• Prompt variation.
• Groundedness.
• Hallucination.
• Unsafe output.
• Prompt injection.
• Data leakage.
• Tool permission boundaries.
• Model/version regression.
• Non-deterministic output tolerance.

Do not test generative output like deterministic software

Too brittle

assert response == "The exact expected sentence"

Prefer properties

assert required facts are present assert forbidden claims are absent assert output matches schema assert citations support claims assert sensitive fields are not exposed assert response is within acceptable latency

Build an AI evaluation dataset

• Common user requests.
• Difficult examples.
• Ambiguous input.
• Long context.
• Missing information.
• Adversarial input.
• Sensitive-data scenarios.
• Known historical failures.

Run this dataset whenever prompts, models, retrieval, tool definitions or system instructions change.

Prompt, RAG, fine-tune or agent?

Need	Start with
Simple transformation or drafting	Prompting.
Private/current knowledge	RAG.
Highly consistent specialised behaviour	Prompting, then consider fine-tuning.
External calculation or factual lookup	Tool use.
Multi-step actions across systems	Agent workflow.
New foundation model	Pretraining only when exceptional need/resources justify it.

Common generative AI mistakes

Mistake	Better approach
Using an LLM for fixed business rules	Use deterministic code where rules are known.
Assuming fluent output is correct	Verify important claims.
Fine-tuning to add current knowledge	Consider retrieval first.
Sending excessive context	Retrieve only relevant information.
Building before defining evaluation	Create representative examples early.
Giving an agent broad production access	Use least privilege and approval gates.
Ignoring model changes	Regression-test prompts/models/retrieval.
Measuring only usage	Measure quality, cost, latency and business outcome.

Generative AI project checklist

• The user problem is clearly defined.
• An LLM is appropriate for the task.
• Representative evaluation examples exist.
• A simple prompt baseline has been tested first.
• Required factual knowledge is identified.
• RAG is considered when knowledge is private or changing.
• Fine-tuning is justified by a measured behaviour gap.
• Model choice is based on task-specific evaluation.
• Latency and cost targets are defined.
• Input and output validation exists.
• Personal/confidential data handling is documented.
• Prompt-injection and tool-use risks are assessed.
• Agents use least-privilege permissions.
• High-impact actions have approval gates.
• Logs and monitoring support investigation.
• Prompt/model/retrieval changes are regression-tested.
• Users understand important limitations.
• Human escalation exists for uncertainty or failure.

Generative AI cheat sheet

Concept	Remember
LLM	Generates language from instructions and context.
Prompting	Tell the model what to do and provide context.
RAG	Retrieve relevant external knowledge before generation.
Fine-tuning	Adapt model behaviour with additional training examples.
Pretraining	Large-scale foundational model training.
Instruction tuning	Train models to follow instructions better.
RLHF	Use human preferences as part of model post-training.
Tool use	Let the model invoke external capabilities.
Agent	AI workflow that chooses and executes multiple actions toward a goal.
Evaluation	Measure behaviour on representative inputs.
Responsible AI	Manage risk, safety, privacy and human impact throughout the lifecycle.

Key takeaways

• Generative AI is a general-purpose technology for creating and transforming text, code, images and other content.
• LLMs are powerful but probabilistic; correctness must be verified according to risk.
• Clear prompts, context, constraints and examples materially improve results.
• Start a generative AI project with a simple baseline and an evaluation set.
• Use RAG for changing/private knowledge and fine-tuning for persistent specialised behaviour.
• Pretraining a foundation model is rarely necessary for ordinary business applications.
• Tool use and agents expand AI from content generation to actions.
• Analyse automation at the task level rather than assuming whole jobs disappear.
• Responsible AI requires security, privacy, monitoring and meaningful human oversight.
• QA engineers can contribute not only by using AI, but by designing the evaluation and controls that make AI systems dependable.

Useful links

• Generative AI for Everyone ↗ — foundational generative AI concepts and applications.
• NIST Generative AI Profile ↗ — risk-management guidance for generative AI.
• NIST AI Risk Management Framework ↗ — broader framework for trustworthy AI risk management.