Principal AI Engineering Architect

Principal AI Engineering Architect

We're looking for a Principal AI Engineering Architect to lead the design and delivery of complex, multi-domain systems spanning cloud, data, and AI - with deep, hands-on mastery of multi-agent agentic AI solutions. This role is ideal for a deeply experienced engineer who owns the hardest architectural challenges, sets technical direction, and serves as the senior technical voice on the engagements they support, while also being able to roll up their sleeves and lead model development, agentic system design, and production delivery end-to-end. In this role, you will operate as the senior technical authority on a cross-functional team, defining architecture across cloud infrastructure, data platforms, and AI/ML workloads, with a strong bias toward AWS-native services and AWS GenAI offerings. You'll partner closely with leadership and clients on technical strategy, lead the design and delivery of complex, production-grade multi-agent systems, mentor experienced engineers, and own high-stakes decisions that shape the long-term success of the systems you build.

We ship production-ready AI in 30 to 45 days. That pace demands people who take ownership, lead with craft, and care deeply about what they put their name on.

Why This Role Matters

At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate. Every role here contributes directly to that mission.

What You'll Do

Craft & Delivery

• Define technical strategy and lead architectural design across cloud, data, and AI/ML systems for end-to-end engagements, owning architecture decisions and driving solutions from research through production at scale
• Architect and ship production-grade multi-agent agentic AI systems, including agent orchestration, tool use, memory, and inter-agent communication patterns
• Design and build with Amazon Bedrock AgentCore and complementary AWS GenAI services to deploy, scale, and operate agentic workloads securely in production
• Architect scalable cloud-native solutions with a strong bias toward AWS, including multi-cloud and hybrid strategies where needed (AWS primary, with Azure, GCP, Kubernetes as secondary)
• Design data architectures including warehouses, data lakes, and pipelines for batch and streaming workloads (e.g., Snowflake, Redshift, BigQuery, Spark, Kafka)
• Design AI/ML systems including model serving, MLOps pipelines, feature stores, and LLM-based applications (e.g., SageMaker, Bedrock, AgentCore, Vertex AI, MLflow, Hugging Face)
• Build and evolve scalable ML platforms, pipelines, and infrastructure that support reliable, repeatable model development and deployment across teams
• Define infrastructure as code, CI/CD, and DevOps standards across engagements (e.g., Terraform, CloudFormation, GitHub Actions)
• Drive performance, scalability, cost, and reliability optimization across deployed systems
• Ensure architecture meets security, governance, and compliance requirements (e.g., GDPR, HIPAA, SOC2)
• Lead cloud migrations and platform modernization initiatives
• Set the standard for AI-forward engineering, using tools like Claude and Cursor with sophistication and helping the team adopt them effectively

Collaboration & Communication

• Partner with senior leadership and clients as the principal technical voice on strategy and direction
• Translate complex AI tradeoffs, risks, and opportunities into clear narratives that drive decision-making across technical and non-technical stakeholders
• Lead design reviews and technical discussions, raising the bar for engineering rigor and constructive challenge across the team
• Engage closely with engineering, data, AI, and product teams to align architecture with broader business priorities
• Develop and maintain architecture documentation, standards, and guidelines

Leadership & Influence

• Define and champion architectural standards and best practices across the engagements you support, bringing depth on tradeoffs, long-term implications, and responsible AI practices
• Mentor and grow engineers at all levels, multiplying impact through coaching, code reviews, and pairing on hard problems
• Own the most difficult architectural, integration, and agentic-system challenges, serving as the senior technical decision-maker and driving them through to production with care for reliability, cost, and safety
• Evaluate emerging technologies - especially in the agentic AI and AWS ecosystems - and recommend tools, frameworks, and patterns that improve architecture over time

What You'll Bring

• 8+ years of software engineering experience, with at least 5 years in technical leadership roles and 4+ years focused on AI/ML systems in production
• Expert software engineering background (Python or similar) with strong design sensibilities for scalable, maintainable systems
• Deep, hands-on expertise designing and shipping production multi-agent agentic AI systems, including agent orchestration, planning, tool use, and multi-agent coordination patterns
• Deep expertise with AWS, including in-depth knowledge of AWS GenAI offerings and hands-on experience with Amazon Bedrock AgentCore; broader multi-cloud experience (Azure, GCP) is a plus
• Strong background in microservices, serverless, containers, and event-driven systems (e.g., Kubernetes, Docker, Lambda, EventBridge)
• Proficiency with infrastructure as code and CI/CD (e.g., Terraform, CloudFormation, Pulumi, GitHub Actions)
• Strong data architecture expertise across relational, NoSQL, and big data systems (e.g., PostgreSQL, MongoDB, Snowflake, BigQuery, Spark, Kafka)
• Hands-on experience with data modeling, ETL/ELT pipelines, and orchestration (e.g., Airflow, Prefect, dbt)
• Mastery of AI frameworks and orchestration tools for building agentic systems (e.g., LangChain, LangGraph, AgentCore, CrewAI, AutoGen, or equivalents)
• Strong experience designing AI/ML systems for production, including LLMs, MLOps, and model serving (e.g., SageMaker, Bedrock, Vertex AI, MLflow, Hugging Face, PyTorch, TensorFlow)
• Strong experience with evaluation frameworks and observability tools for LLM and agentic apps, including building these capabilities where they don’t yet exist
• Deep understanding of AI safety, responsible AI principles, prompt injection defenses, and PII handling
• Extensive experience building RAG pipelines: chunking strategies, embedding models, vector databases, and advanced retrieval techniques
• API design experience, including architecting and integrating with internal and third-party services at scale
• Advanced cost optimization expertise: token economics, caching strategies, model routing, quantization
• Solid understanding of networking, security, identity, and access management in cloud environments
• Experience with governance, compliance, and observability frameworks
• Track record of senior technical leadership and mentoring experienced engineers
• Strong stakeholder communication skills, with the ability to translate technical depth across audiences
• Demonstrable, day-to-day usage and expert knowledge of AI-forward coding tools such as Claude Code and Cursor
• Multi-cloud architecture experience, AI ethics or responsible AI experience, or enterprise architecture certifications (e.g., TOGAF, AWS/Azure/GCP) is a plus

Our salary range is $180,375 - $230,625 USD