• Build and maintain the eval platform, including harnesses, datasets, graders, and CI integration
• Measure agent behavior through task success, tool-use correctness, drift, latency, and cost
• Perform hands‑on exploratory end‑to‑end testing based on how members, operations teams, and providers use the product
• Work with business stakeholders to understand real workflows and validate that the right product was built
• Develop domain expertise in Curative’s business workflows
• Use traces, failure taxonomies, and dashboards to turn agent issues into specific findings
• Reproduce production failures, identify root causes, fix issues, or provide precise diagnoses
• Advise on quality concerns and make product behavior or requirements decisions when needed
• Prioritize testing and quality work according to risk and business impact
• Review AI-generated code changes and maintain quality standards
Requirements
- 5+ years in software quality, testing, or engineering with real range
- Experience writing automation and performing serious exploratory testing
- Fluency with LLM evaluation, including golden datasets, LLM-as-judge, programmatic graders, and regression suites for prompts and agent loops
- Hands‑on experience with AI agents and failure modes including silent drift, tool misuse, and compounding errors
- Debugging depth with logs, traces, queries, and production forensics
- Ability to fix bugs in Python or TypeScript
- Ability to work with non‑technical business stakeholders and operations leads
- Comfort with ambiguity, speed, and working without mature processes or sprint boundaries
- Pragmatic quality judgment
- Sharp written communication
- Claude Code, Cursor, or equivalent as a primary development tool
- Uses AI to accelerate quality work while recognizing the limits of AI judgment
- Reviews every AI‑generated diff and does not merge on vibes
- Strongly preferred: experience standing up eval or observability infrastructure from zero
- Strongly preferred: LLM observability tooling such as LangSmith, Braintrust, Arize, or homegrown tools
- Strongly preferred: defining unsupervised agent capabilities and building guardrails
- Strongly preferred: product requirements and behavior definition experience
- Strongly preferred: building deep domain expertise in a complex operational business
Core Competencies
Demonstrates expertise in software quality assurance, including automation, exploratory testing, and debugging in Python or TypeScript. Possesses strong communication skills to collaborate with business stakeholders and develop domain knowledge in complex operational workflows.
Highest‑signal resume keywords
- Software Quality Assurance
- Automation Testing
- LLM Evaluation
- Debugging in Python
- AI Agent Experience
ATS Optimization Keywords
Hard Skills
- Software Quality
- Exploratory Testing
- Automation Writing
- Debugging Depth
- Production Forensics
- Failure Modes Analysis
- Root Cause Identification
- Quality Judgment
- Agent Behavior Measurement
- Product Requirements Definition
Soft Skills
- Sharp Written Communication
- Comfort with Ambiguity
- Pragmatic Quality Judgment
Industry Keywords
- Eval Platform
- Observability Infrastructure
- AI‑Generated Code Review
- Business Workflows
- Agent Capabilities
Tools & Technologies
- Claude Code
- Cursor
- LLM-as-Judge
- LangSmith
- Braintrust
- Arize
#J-18808-Ljbffr
Senior Software Test Engineer Arbeitgeber: Jobtailor
Als Front Office Supervisor in unserem dynamischen Team bieten wir Ihnen die Möglichkeit, in einem unterstützenden und freundlichen Arbeitsumfeld zu wachsen. Wir legen großen Wert auf die berufliche Entwicklung unserer Mitarbeiter und bieten regelmäßige Schulungen sowie die Chance, Verantwortung zu übernehmen. Unsere Lage ermöglicht es Ihnen, Teil einer lebendigen Gemeinschaft zu sein, während Sie gleichzeitig die Standards unseres Franchise-Partners einhalten und unseren Gästen einen unvergesslichen Aufenthalt bieten.
