Senior QA Engineer in Berlin

Your mission

A senior, hands-on engineering role responsible for creating measurable confidence in both the conventional software platform and the agentic AI workflows behind it. You'll test the complete journey from incoming emails and files through extraction, material matching, RFQ creation and quotation - verifying not only that the software executes correctly, but that the resulting business data and AI decisions are accurate, safe and commercially sensible. The objective is to turn quality from manual judgment into an automated, evidence-based release process. Your gates should answer: does the application work, are the integrations behaving, did the AI extract the right information, did the agent choose the right tool with valid arguments, is the business outcome correct, has this change regressed anything, and is the release safe to deploy.

WHAT YOU'LL OWN

• Test strategy. Define and own the testing strategy across the platform - what gets tested at which layer (unit, component, contract, integration, end-to-end, production verification), where deterministic assertions end and statistical evaluation begins, and how quality gates map to release risk. Keep it a living document engineers actually use, not a compliance artifact.
• Software test automation. Automated coverage across the React web app, backend APIs and services, auth and authorization, database operations, third-party integrations, email and document ingestion, background jobs and async workflows, multi-tenant configuration and isolation, and end-to-end RFQ and quotation journeys. Playwright, pytest, API clients, SQL, CI/CD. Focus on critical business journeys and high-risk failure modes, not test-case count.
• API and integration testing. Beyond happy-path responses: request and response schemas, permissions, invalid and incomplete inputs, error handling, idempotency and duplicate requests, timeouts and retries, async state transitions, webhooks and event processing, data consistency across services, tenant isolation, and integration failure and recovery - running in CI with enough diagnostic output to explain a failure.
• AI and LLM evaluations. Versioned evaluation datasets built from real business scenarios, edge cases and observed failures. Targets include extraction accuracy, material and product matching, classification and routing, agent decision-making, generated RFQ and quotation content, grounding, hallucination frequency, instruction adherence and final business correctness. You define the scoring methods - exact assertions, semantic comparison, rule-based and model-based graders, human review where needed. Because behaviour is nondeterministic, expect quality thresholds and statistical comparison rather than identical output every run.
• Agent workflow testing. Inspect the whole execution path, not the final response: was the correct tool selected, were its arguments valid, were calls made in the right order, was the right source data used, were permissions and tenant boundaries respected, did retries and recovery work, did the agent stop or escalate when uncertain, did it avoid unnecessary actions, did the outcome match the business objective, and were latency and cost acceptable.
• Controlled use of AI testing agents. Use bounded agents to analyse requirements, suggest missing scenarios, generate candidate tests, explore unusual journeys, create adversarial inputs, reproduce defects and analyse traces. You stay responsible for validating what they generate and for defining authoritative pass/fail criteria - AI output never becomes an uncontrolled release decision.
• Release readiness. Define and maintain the criteria: critical journeys pass, API contracts stable, no cross-tenant access violations, extraction accuracy above the agreed threshold, no material regression in agent task completion, known critical scenarios still succeeding, latency and cost within limits. You communicate what passed, what failed, what's uncertain and the risk of proceeding, and give a recommendation - final release ownership stays shared with engineering and product.
• Failure investigation. Work a regression across layers - browser traces and screenshots, API requests, application logs, agent execution traces, database queries, background-job status, model outputs and grader results, deployment and config changes - and turn every meaningful production failure into a reproducible test or evaluation case.

WHAT YOU'LL PRODUCE
Automated UI and API suites · integration and end-to-end scenarios · version-controlled AI evaluation datasets · agent-workflow graders · CI/CD quality gates · test and evaluation reports · release-readiness dashboards · regression and failure classifications · defect reproduction packages · quality-risk assessments · a living test strategy documenting layer ownership, coverage standards and evaluation methodology.

Your profile

• 5+ years in SDET, quality engineering, test automation or software engineering.
• Strong TypeScript or Python, with experience designing maintainable test frameworks - not only writing test cases.
• Strong Playwright, pytest or comparable automation experience, and advanced API and integration testing knowledge.
• Experience defining test strategy for a platform or organisation - deciding what gets tested where and why, not only implementing someone else's plan.
• Experience testing asynchronous and distributed systems.
• SQL, logging and production-debugging skills; CI/CD and automated release gates.
• Strong judgment when requirements or outcomes are ambiguous - and the ability to distinguish technical correctness from business correctness.
• EU work permission.

USEFUL ADDITIONAL EXPERIENCE

• LLM and agent evaluations, trace-based debugging, model-based grading, prompt-injection and adversarial testing.
• Multi-tenant SaaS platforms; email and document processing.
• Procurement, manufacturing or supply-chain systems; data-quality and matching problems.

About us

We are a Berlin-based startup that has successfully established a leading digital-first trading company for industrial metals across Europe.

The multi-billion-euro metal trading industry is operated on Excel, PDF and Email. Our mission is to integrate smart technology and data to drive every decision, transaction, and workflow in the steel industry. Our technologies & services increase liquidity, accelerate transactions, reduce scrapping rates and enhance trading convenience for hundreds of steel and metal distributors across Europe.