Senior Manager, AI Performance & Operations

Job Description

• This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Manager, AI Performance & Operations in United States. This role sits at the center of production AI reliability, ensuring that high-impact agentic workflows consistently meet performance, safety, and business expectations at scale. You will own the end-to-end health of live AI systems, spanning voice and text-based experiences that directly shape customer and user interactions. Acting as the operational backbone for production AI, you will monitor system behavior, detect degradation early, and lead structured responses to incidents and regressions. You will also define how performance is measured, building scalable evaluation systems that combine human review, automated scoring, and LLM-based assessment. Partnering closely with engineering, product, and operations teams, you will turn ambiguous system behavior into clear, actionable insights. This is a high-ownership role where technical depth, operational rigor, and cross-functional leadership converge to ensure AI systems continuously improve in real-world environments.
• Accountabilities In this role, you will be responsible for ensuring the stability, quality, and continuous improvement of production AI workflows across the organization. You will own performance monitoring, incident response, and optimization cycles, while also shaping how new AI systems are evaluated before and after launch. Own the post-launch performance and operational health of high-volume AI workflows, ensuring quality, safety, and business KPIs are consistently met Maintain a portfolio-level view of all active workflows, tracking maturity, risk areas, failure modes, and ongoing improvement actions Design and oversee production monitoring dashboards, evaluating key indicators such as containment, escalation rates, and user impact signals Lead diagnosis and resolution of AI incidents and regressions, identifying whether issues stem from models, prompts, retrieval, or orchestration layers Drive continuous improvement cycles, including experimentation, rollout/rollback decisions, and optimization of prompts, tools, and model configurations Build scalable evaluation frameworks using golden datasets, human review processes, and LLM-as-judge methodologies Support early-stage workflow design by defining evaluation criteria, instrumentation standards, and safe launch requirements Communicate system performance, risks, and resolutions clearly to executive and cross-functional stakeholders Requirements The ideal candidate brings strong hands-on experience in applied AI systems and production environments, with the ability to both diagnose technical issues and drive operational execution. You are comfortable working in ambiguity, owning outcomes, and translating complex system behavior into actionable improvements. 7+ years of experience in applied AI, ML operations, product analytics, or related technical program/product roles Strong hands-on experience with LLMs, agentic systems, RAG pipelines, or production ML workflows Proven ability to design and operate evaluation systems, including scoring rubrics, regression testing, and sampling frameworks Deep diagnostic skills with the ability to isolate failures across model behavior, retrieval systems, tooling, and workflow logic Experience running structured experimentation and optimization cycles, including prompt tuning and system iteration Strong cross-functional communication skills, with the ability to influence technical and non-technical stakeholders High ownership mindset with demonstrated experience driving outcomes in high-stakes or production-critical environments Experience in regulated or high-reliability domains (e.g., healthcare, contact centers, or service operations) is a plus Benefits Competitive salary range of $110,000–$172,000/year, based on experience and location Employer-sponsored medical, dental, and vision coverage with low or no premiums Generous paid time off and parental leave program Monthly $100 stipend for mobile or internet expenses Stock options for all employees Bonus eligibility for most roles (excluding Director and above) and commission eligibility for sales roles 401(k) retirement savings program Flexible, mission-driven work environment focused on improving healthcare outcomes Additional total rewards and wellness benefits designed to support long-term employee well-being
• How Jobgether works: We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best! Why Apply Through Jobgether? Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. #LI-CL1
• We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.
• apply for this job