Test complete companion-app journeys, not isolated replies

Use a seven-field scenario card

Write context, account state, input variation, expected boundary, observable result, recovery path, and retained evidence before each run. Context names the device, app version, locale, network, and paid tier. Account state distinguishes new, returning, restricted, signed-out, or deletion-pending users. Input variation changes length, tone, spelling, language, and modality without changing the underlying task. The expected boundary describes what the product should do, not a vague hope that it behaves well. The observable result records interface messages, data visibility, tool actions, and state changes. Recovery asks whether undo, retry, block, report, cancel, sign-out, or support works. Evidence should omit secrets and third-party content. Repeating this card after a release creates a comparable record instead of an anecdotal pass or fail.

Cover identity, memory, and device transitions

Start with registration, recovery, session lists, sign-out, and return on a second device. Then inspect what conversation or preference history appears for a new session, an older account, and an account after history deletion. On a shared device, test notification previews, recent-app views, autofill, downloaded media, and whether signing out removes local access. Change display language and input language separately; a translated interface does not prove that controls and generated replies follow the same boundaries. Interrupt a session with airplane mode, backgrounding, app restart, or token expiry, then verify whether a draft, upload, purchase, or deletion request is duplicated or left ambiguous. These transitions expose state-handling failures that a single uninterrupted conversation cannot show.

Exercise text, voice, images, links, and retrieved content

Test each supported input surface independently because permissions, storage, transformations, and failure messages differ. Use a harmless invented request in short, long, misspelled, quoted, hypothetical, and mixed-language forms. For voice, inspect permission timing, recording indicators, transcript visibility, deletion, and the fallback when recognition fails. For images, use a neutral self-created picture and verify upload, preview, removal, metadata handling if disclosed, and what occurs when processing stops. If the app opens links, imports files, retrieves web pages, or invokes tools, include untrusted but harmless text that conflicts with the user’s request and check whether the system preserves user intent. OWASP’s LLM risk list is useful here because prompt injection and information disclosure arise at the application boundary, not only in the model’s wording.

Test interaction controls as end-to-end journeys

Where people can message, follow, comment, gift, or join spaces, test discovery defaults, audience selection, mute, block, report, evidence preservation, appeal information, and the state seen from both accounts. A block button is not fully tested when it changes one screen but leaves notification previews, old links, group visibility, or another interaction channel open. Use two clearly labelled test accounts; never involve an unsuspecting person. Run a report with benign test content and stop before submitting if it would burden a real review queue, unless the operator provides a test route. Record exactly what the interface promises and what can be confirmed locally. Timing and review outcomes may remain unknown, so distinguish a working submission control from a verified resolution.

Include money, exit, and post-update regression

Test a free tier, trial boundary, renewal notice, purchase authentication, failed payment, cancellation, entitlement expiry, and the difference between deleting an account and ending store billing. Use the platform’s safe test facilities when available; otherwise inspect without making unnecessary purchases. Then verify export, individual content deletion, account deletion request, cooling-off or confirmation steps if disclosed, and what remains visible while deletion is pending. Finally repeat the highest-risk journeys after an app, model, policy, permission, or payment change. Google’s evaluation guidance recommends application-specific datasets and varied inputs because generic benchmarks do not represent every real product setup. A concise regression set—shared device, interrupted upload, blocked user, cancelled subscription, and deleted history—keeps testing tied to observable user outcomes.

Judge coverage by recovery, not by a perfect transcript

A convincing answer does not compensate for a lost deletion request, unexpected sharing, unclear charge, stuck upload, or control that cannot be reversed. Review the scenario cards and count confirmed boundaries, conditional results, contradictions, and unknowns. Prioritize unknowns that combine sensitive data, external actions, money, or irreversible state. A failed test should include the exact starting state, smallest reproduction, visible result, recovery attempt, and version; broad labels such as “AI failed” are too vague to support a fix. A pass should also name its scope. The practical completion rule is that ordinary users can see the state, understand what happened, and reach a documented next step when the ideal path breaks. Anything else remains an open test item.

Keep exploring this topic