Director of AI Operations – Governance

• Evaluate, benchmark, fine-tune, and adjust configurations of ML and generative AI models (including LLMs) in production, applying working knowledge of model training, tuning, and evaluation methodologies rather than relying solely on vendor documentation.
• Apply applied AI/ML research and emerging techniques to inform build-vs-buy decisions, model selection, and architecture choices across the AI portfolio.
• Partner directly with data science and ML engineering teams on model performance issues, drift detection, and retraining or reconfiguration needs, contributing technical judgment rather than acting purely as a liaison.
• Maintain technical fluency in prompt engineering, retrieval-augmented generation, agentic/orchestration frameworks, and the practical distinctions between generative AI and traditional ML systems, and translate those distinctions into governance and staffing decisions.
• Own AI sustainment and governance for workplace AI tools across enablement, bought solutions, and custom builds, acting as the central \"run\" function for the AI strategy, and building the team and processes to scale it.
• Manage all AI-related licenses and entitlements: monitor usage, optimize allocations, drive reallocation, and partner with Finance for cost visibility and optimization.
• Monitor production usage patterns and performance to recommend roadmap items, enhancements, and deprecations based on real-world dynamics in production.
• Own and triage support tickets for workplace AI tools and platforms, driving resolution across vendors, internal engineering, and security partners.
• Lead continuous evaluations of AI tools and models, including monitoring drift, benchmarking performance, and ensuring tools remain effective and aligned with business KPIs.
• Maintain the updates and security outlook for AI platforms, coordinating patches, version upgrades, vulnerability remediation, and compliance with Shield AI security policies.
• Orchestrate model swaps and configuration changes in production, including rollout planning, risk assessment, change control, and post-deployment monitoring.
• Design, maintain, and govern shared prompt libraries, including standards for prompt quality, reuse, versioning, and training for end-users and builders.
• Own management of secrets (API keys, credentials, tokens) used by AI tools and orchestrators, ensuring secure storage, rotation, and access control in partnership with Security and IT.
• Define and maintain connectors and extensions (e. g., integrations into Saa S systems, data sources, and workflow tools) to ensure reliable, secure data access for AI workflows.
• Establish and operate auditability frameworks for AI tools, including logging, traceability of AI‑assisted actions, and reporting for compliance and risk management.
• Lead AI governance practices for workplace AI (policies, guardrails, usage standards, approval workflows, exception processes) in partnership with Security, Legal, and HR.
• Partner with business solution and build teams to ensure their deliverables meet sustainment, observability, and governance requirements before moving to production.
• Define operational playbooks, SLAs, and incident response procedures for AI systems, including on-call patterns supported by contractors and platform specialists.

Requirements

• 15+ years in platform operations, ML/AI operations, Dev Ops, or Saa S sustainment roles, including significant experience in leadership/people management, with a track record of running production systems in a high-stakes environment (defense, aerospace, enterprise Saa S, or similar).
• Direct, hands-on experience developing, training, fine-tuning, or evaluating machine learning models or generative AI systems — this is a core requirement, not a nice-to-have.

Candidates should be able to speak credibly to model architecture, training/tuning approaches, and evaluation methodology.

• Working knowledge of AI/ML research practices and the ability to apply current research to production decision-making.
• Software engineering or data science background sufficient to engage deeply with technical teams on model behavior, integration issues, and system design tradeoffs.
• Direct experience with AI platforms or orchestration tools (e. g., LLM providers, RPA/workflow tools like n8n, enterprise Saa S integrations) and their operational management, including at an organizational or strategic level.
• Demonstrated understanding of the distinct technical and operational challenges of generative AI versus traditional ML — including differing skill requirements, risk profiles, and market/salary dynamics — and ability to apply that distinction to team design and hiring.
• Strong background in governance, compliance, or security in the context of data-driven or AI systems, including familiarity with audit, logging, and access control best practices.
• Demonstrated ability to manage licenses and cost optimization for Saa S or AI tools at scale, including working with Finance and procurement stakeholders, and managing significant budgets.
• Hands‑on experience with monitoring and observability stacks (logs, metrics, alerts) and using those signals to shape product roadmaps and operational improvements.
• Strong technical fluency across APIs, connectors, and integrations; able to work closely with engineering and vendors to design and maintain extensions.
• Proven track record building and leading high-performing teams, including hiring, mentoring, and developing talent across a mix of core staff and contractors.
• Excellent executive communication skills, with ability to translate production dynamics and risk into clear recommendations for senior business and technical leaders, including executive stakeholders.
• Experience operating in a hybrid environment of contractors and core team members, with the ability to define processes and standards that scale as the team matures.
• Core Competencies

Demonstrates extensive expertise in AI and ML operations, including model training, tuning, and evaluation methodologies, while effectively managing AI governance, compliance, and cost optimization across platforms.

Proven ability to lead high-performing teams and communicate complex technical concepts to executive stakeholders.

• Highest-signal resume keywords
• Machine Learning Model Development
• Generative AI Systems Management
• AI Governance and Compliance
• Cost Optimization for Saa S Tools
• Team Leadership and Development
• ATS Optimization Keywords
• Hard Skills
• Model Training
• Model Tuning
• Model Evaluation
• AI/ML Research Practices
• Monitoring and Observability Stacks
• API Integration
• Data Science Background
• Risk Assessment
• Change Control
• Audit and Logging Best Practices
• Soft Skills
• Executive Communication
• Team Building
• Mentoring
• Collaboration
• Problem Solving
• Industry Keywords
• Platform Operations
• Dev Ops
• Saa S Sustainment
• High-Stakes Environment
• Data-Driven Systems
• Tools & Technologies
• LLM Providers
• RPA/Workflow Tools
• Saa S Integrations
• AI Platforms
• Orchestration Tools
• #J-18808-Ljbffr