Apple employs thousands of people to review App Store submissions. Someone opens the app, taps around, checks guidelines, writes up findings. It still takes days. That has been the cost of running a platform for as long as platforms have existed.
Jest has the same problem. Every app submitted to our marketplace needs evaluation before it goes live:
- Go through core flows
- Verify SDK initialization
- Confirm user data, notifications, payments, and referrals are wired up
- Catch JavaScript errors and broken UI states
- Write a summary
Despite being live only for 6 months, we have had thousands of app build submissions. Each one requires a full manual playthrough.
Human reviewers carry blind spots. They skip flows they assume work. They miss edge cases on their third review of the afternoon.
We are a small team. Throwing bodies at this was never going to hold. So we built Player 01: our AI Agent designed to autonomously test all Jest app submissions.
The Agentic Approach
Unlike traditional automated testing scripts that follow a rigid path, Player 01 is an exploration agent. It doesn’t just run a test suite; it “sees” the app and decides how to interact with it in real-time.
The infrastructure spins up a headless Chromium instance via Playwright, configured to a 390x844 mobile viewport. It creates a temporary sandbox session so the agent runs fully authenticated, identical to what a real user sees. Before the page loads, we inject an SDK interceptor script. Every SDK call between the app and the Jest host gets timestamped and logged in both directions.
The “brain” of the agent is a custom executor bridging Anthropic’s computer use capability to Playwright. Player 01 perceives the screen, reasons about the UI, and executes actions (click, type, scroll, keypress) that map to real browser interactions.
What makes this an agent rather than a tool is its recovery logic. If Player 01 stalls or the app hangs, a secondary reasoning process takes over: it reviews the recent recording, formulates a new plan to bypass the obstacle, and re-enters the loop with more targeted instructions. When the mission is complete, it offloads the SDK call logs and session data to our analysis pipeline.
Automated Synthesis
Raw exploration data tells you what happened. We wanted the agent to tell us what matters.
Once Player 01 finishes its session, we push the data through two more AI passes. The first evaluates the app’s external behavior against our integration checklist. By monitoring the traffic between the app and the browser, the agent verifies required items (SDK init, host communication) and monitors critical flows like user data requests, notification scheduling, and payment links.
The second pass generates a written summary. The model reads the captured SDK calls, the sequence of agent actions, and the resulting screenshots to produce a report. Instead of just a simple “Pass” or “Fail,” it provides a narrative of the experience: what it successfully tested, where it saw inconsistencies in the UI, and where the SDK logs didn’t match the expected app state.
The Reviewer Experience
Everything lands on a single report page in our management console:
- Agent Summary: A written overview of the app’s quality and integration concerns.
- SDK Checklist: Pass/fail for every monitored integration point.
- Exploration Timeline: A unified chronological view of agent actions and SDK calls.
- Video Playback: The full session at up to 5x speed.
- Screenshots: Every visual state the agent captured.
A human reviewer now skims the agent’s findings, examines the checklist, and only drills into the timeline if a red flag appears. 20 minutes of manual testing has been condensed into 2 minutes of informed review.
Player 01 does not unilaterally approve or reject apps. It acts as a force multiplier, giving our team better information, faster.
What’s Next
Player 01 now runs automatically for every app submission. While this currently streamlines our internal workflow, our ultimate goal is to open up these agent reports directly to developers. By sharing these automated reviews, we can provide instant, actionable feedback the moment a build is uploaded - helping devs squash bugs and polish integrations faster than ever.
If you build on Jest, this means shorter feedback loops and a dedicated AI partner to help you ship a better app.
If you’re curious about the future of AI-driven testing or want to publishing a messaging app on Jest, contact us or check out the Jest Apps Fund, which provides up to $1M in growth capital for devs building on Jest.