We’re building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you’ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time.
Overview
We are seeking a Principal Individual Contributor (IC) to lead production engineering, observability, and operational excellence for our AI Platform. This role sits at the intersection of ML systems, distributed infrastructure, and production reliability, ensuring that our AI services are scalable, observable, and resilient in real-world environments.
As a senior technical leader, you will define and drive best-in-class production practices, build robust monitoring and alerting ecosystems, and partner across engineering, ML, and platform teams to ensure mission-critical AI systems meet high availability, performance, and reliability standards.
Key Responsibilities
Production Reliability & Operations Leadership
Own and evolve production operations strategy for AI/ML platforms and services
Define SLOs, SLIs, and error budgets for AI systems (online & batch/inference pipelines)
Lead root cause analysis (RCA) and drive systemic improvements post-incident
Establish operational readiness standards for launching new AI capabilities
Build frameworks for on-call excellence, incident response, and escalation
Observability, Monitoring & Alerting
Design and implement end-to-end observability systems across AI workloads:Model performance monitoring
Data pipeline health
Infrastructure metrics
Build and scale monitoring and alerting frameworks using modern tooling (e.g., Prometheus, Grafana, OpenTelemetry, Datadog, Azure Monitor, etc.)
Define actionable, low-noise alerts tied to business and system impact
Develop dashboards and telemetry standards for real-time visibility across services
Drive adoption of golden signals (latency, errors, throughput, saturation) in AI systems
AI/ML Production Systems Excellence
Ensure reliable deployment and operation of:Real-time inference services
Model pipelines (training, validation, deployment)
Data ingestion and feature pipelines
Implement model observability (drift detection, data skew, performance degradation)
Partner with ML engineers to improve production readiness of models
Establish lifecycle standards for models in production environments
Automation & Platform Development
Build internal platforms and tooling for:Automated incident detection and response
Self-healing systems
Deployment validation and canarying
Drive Infrastructure as Code (IaC) and policy automation
Improve system resilience through chaos testing and fault injection
Technical Leadership & Strategy
Act as a trusted technical advisor across platform, ML, and product teams
Set direction for operational excellence in AI systems at org scale
Mentor senior engineers and influence cross-team architectural decisions
Lead adoption of industry best practices in reliability engineering and observability
Required Qualifications
10+ years in software engineering, production engineering, or SRE roles
Deep experience operating large-scale distributed systems in production
Proven track record building monitoring, observability, and alerting systems
Strong expertise in incident management and production support models
Experience working with cloud platforms (Azure, AWS, GCP)
Preferred Qualifications
Experience supporting AI/ML platforms or data-intensive systems
Familiarity with model lifecycle management and MLOps practices
Knowledge of:OpenTelemetry, Prometheus, Grafana, Datadog
Kubernetes and containerized workloads
Streaming systems (Kafka, Event Hub, etc.)
Experience defining and implementing SLO-driven engineering
Background in high-availability, low-latency systems
Key Competencies
Systems thinking and ability to reason about complex, interdependent systems
Strong bias for automation, scalability, and long-term solutions
Exceptional debugging and incident management skills
Ability to influence without authority across multiple teams
Passion for operational excellence and reliability
Pay Range
The typical pay range for this role is:
$144,200.00 - $288,400.00
This pay range represents the base hourly rate or base annual full-time salary for all positions in the job grade within which this position falls. The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors. This position is eligible for a CVS Health bonus, commission or short-term incentive program in addition to the base pay range listed above. This position also includes an award target in the company’s equity award program.
Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong.
Great benefits for great people
We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families.
This full‑time position is eligible for a comprehensive benefits package designed to support the physical, emotional, and financial well‑being of colleagues and their families. The benefits for this position include medical, dental, and vision coverage, paid time off, retirement savings options, wellness programs, and other resources, based on eligibility.
Additional details about available benefits are provided during the application process and on Benefits Moments.
We anticipate the application window for this opening will close on: 06/04/2026
Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state and local laws.