Quality Assurance Engineer

About Napster

Napster has always been about democratizing access: first to music in 1999, and today to expertise.

We are a frontier AI company powering the next generation of embodied and agentic AI, building lifelike AI agents with voice, video presence, persistent memory, and real-time utility across apps, websites, physical spaces, and workplaces.

Our platform, including the Napster Omniagent API, our real-time conversational video models, and hardware like Napster Station and Napster View, lets organizations deploy AI coworkers, digital twins, and specialist agents that engage people face to face.

Our team is a fast-growing group of engineers, researchers, designers, and business leaders working together to make AI that is useful, trustworthy, and human in how it connects.

About the role

As a QA Engineer on the Professional Services team at Napster, you will own the quality of the AI agents and experiences we ship to our most strategic customers.

Your core mission is to make quality measurable and automatic: writing AI evals that tell us whether an agent is accurate, on-brand, and safe, building UI test automation for the web, mobile, and kiosk experiences around it, and wiring both into the pipelines our Forward Deployed Engineers ship through every day.

Napster's products are hard to test with traditional methods alone.

Our agents speak, see, remember, and use tools in real time, often in multiple languages and inside customer systems.

You will design how we test that, whether it's a digital twin of an executive greeting shoppers, a multilingual agent in a public-sector kiosk, or a crew of specialist agents working inside enterprise workflows.

Automation comes first, but you will also roll up your sleeves for hands-on exploratory and acceptance testing when a deployment needs it, including on real hardware and, occasionally, on site.

Working closely with Forward Deployed Engineers, Solution Engineering, Enablement, Product, and Engineering, you'll be the person who can say with evidence that a deployment is ready for production.

How we build

Our QA work is AI-native.

You'll use agentic coding tools like Claude Code, Cursor, and Codex as everyday teammates to generate test cases, write and maintain automation, triage failures, and build eval datasets in hours rather than weeks.

We treat tests and evals as product: versioned, reviewed, and packaged into reusable agent skills, MCP servers, and templates so every deployment starts with a stronger safety net than the last.

If you've used an LLM to judge another LLM, have opinions about flaky tests, and believe a good eval suite is the fastest way to ship AI with confidence, you'll feel at home here.

Responsibilities

• Design, build, and maintain AI evaluation suites for Napster agents and digital twins, covering answer accuracy, grounding in customer knowledge, tool and function-call correctness, persona and brand fidelity, conversation quality, multilingual behavior, and safety guardrails.
• Build golden datasets and test cases with customers and FDEs, including adversarial, edge-case, and red-team prompts, and turn real conversation transcripts into regression sets.
• Implement LLM-as-judge and rubric-based scoring, calibrate judges against human review, and track quality metrics over time so model, prompt, or knowledge changes never ship as silent regressions.
• Build and own UI test automation for customer-facing experiences across web, mobile, and kiosk front ends, using frameworks such as Playwright, Cypress, or Appium.
• Automate end-to-end and API tests for agent integrations, MCP servers, tool calls, and third-party systems, including latency and reliability checks for real-time voice and video sessions.
• Integrate evals and automated tests into CI/CD so every deployment has clear, automated quality gates and readable reports for the team and the customer.
• Run manual exploratory, acceptance, and device testing when needed, including on Napster Station and Napster View hardware and in physical installations, and write clear, reproducible bug reports.
• Define test strategies and acceptance criteria with Solution Engineering and FDEs at the start of each engagement, and sign off on production readiness.
• Codify repeatable testing patterns into reusable eval templates, test harnesses, agent skills, and plugins, and feed quality insights back to Product and Engineering.
• Contribute to internal programs, including agent dogfooding and enablement, that help the wider team adopt AI-native testing practices.
• Stay current on AI evaluation and testing tools, from eval frameworks and observability platforms to new approaches for testing voice, video, and agentic systems.
• Travel to customer sites for launch testing and acceptance when needed, estimated at up to 15% depending on location.
• Be a champion for quality, reliability, and responsible AI practices in everything we ship.
• You may be a good fit if you have
• 4+ years of experience in QA engineering, SDET, or test automation, ideally including customer-facing, consulting, or professional services work.
• Hands-on experience designing and running evals for LLM-based products, such as golden datasets, rubric scoring, LLM-as-judge, and regression testing of prompts and agents.
• Strong test automation skills in Python and/or Type Script, with a track record of building maintainable UI and API test suites from scratch.
• Production experience with UI automation frameworks such as Playwright, Cypress, Selenium, or Appium, and with integrating tests into CI/CD pipelines (Git Hub Actions, Git Lab CI, or similar).
• Daily use of agentic coding tools such as Claude Code, Cursor, or Codex, and a clear point of view on how to use them to write, maintain, and debug tests.
• Solid understanding of how LLM agents work, including tool use, retrieval, memory, and their typical failure modes like hallucinations, prompt injection, and inconsistent behavior.
• Comfort with manual and exploratory testing, and the judgment to know when a human needs to look at something and when it should be automated.
• High agency and comfort navigating ambiguity in fast-moving product and customer environments.
• Strong communication skills to explain quality risks and test results to developers, customers, and executives, with a low ego and collaborative approach.
• Genuine care about shipping AI experiences that are reliable, safe, and that people actually want to talk to.
• Nice to have
• Experience with LLM eval and observability tooling such as Langfuse, Braintrust, Lang Smith, Arize, Promptfoo, or Deep Eval.
• Experience testing real-time voice, video, or multimodal systems (Web RTC, streaming audio/video, speech-to-text, text-to-speech, avatars), including latency and audio quality measurement.
• Experience with AI red teaming, safety testing, or prompt-injection testing.
• Experience testing hardware or physical deployments such as kiosks, venues, or retail spaces.
• Experience with performance and load testing tools such as k6, Locust, or JMeter.
• Experience building MCP servers, agent skills, or internal test tooling that other engineers use.
• Testing multilingual products, especially Arabic and right-to-left interfaces, or experience working in the Middle East, Europe, or other international markets.
• Background in retail, government and public sector, education, healthcare, or another enterprise vertical.
• Familiarity with enterprise IT, security reviews, accessibility testing, and integration patterns (SSO, CRM, e-commerce, content management).
• Public work that shows how you test: open-source eval suites or test tools, blog posts, demos, or talks.