Skip to content

Advancing Responsible AI Governance: A Comprehensive Model Evaluation Framework for Mount Sinai

Share

How we enabled MSMC to make faster, more confident decisions on AI adoption, improve vendor negotiations, assure, and strengthen patient safety and reliability by implementing a structured evaluation framework.

Client

Mount Sinai Medical Center (MSMC) is a leading healthcare organization recognized for its commitment to innovation and excellence in patient care.   

"Our partnership resulted in a comprehensive and effective AI Model and Assessment tool that strengthens our ability to evaluate AI vendors with confidence. Their expertise, responsiveness, and commitment to quality were instrumental in creating a process that aligns with best practices in AI governance and compliance. "  

Tom Gillette

Chief Information Officer, Mount Sinai Medical Center, FL

a drawing on a paper

Challenge

MSMC is rapidly adopting AI technologies to enhance clinical decision-making, streamline patient communication, and optimize operational workflows. However, as AI models became more embedded in critical healthcare functions, MSMC faced a growing concern: how to ensure these systems were uniformly safe, equitable, and accountable.  

Without a formal AI risk evaluation framework, MSMC encountered several challenges:  

  • Potential bias in AI-driven clinical recommendations, risking unequal treatment outcomes  
  • Patient safety concerns due to unvalidated or opaque model behavior  
  • Legal and reputational risks stemming from AI decisions made without sufficient oversight  

MSMC recognized the urgent need for a structured approach to evaluate, monitor, and govern AI models throughout the implementation lifecycle, ensuring alignment with clinical standards, regulatory expectations, and ethical principles.  

Approach

MSMC worked with EisnerAmper to address growing concerns around AI by implementing a structured, multi-dimensional evaluation framework grounded in seven foundational components. This approach helped rigorously assess all aspects of AI adoption—from governance to monitoring—and align them with MSMC's healthcare standards and core principles.  

 

Evaluation Framework

a group of people walking through a building

Foundational Components with Key Strategies & Considerations  

  • Governance: Defined AI oversight structures, accountability, and policies.  
  • Technology: Assessed data readiness, cloud infrastructure, and security.  
  • Financial: Evaluated total cost of ownership and financial risk.  
  • Clinical Risk & Controls: Integrated SAFER compliance and a risk & safety program.  
  • Model Testing: Developed test plans and success metrics.  
  • Transparency: Enhanced patient communication, usability, and trust.  
  • Monitoring: Implemented real-time surveillance, incident capture, and feedback loops.  
a set of medical tools

10 Categories, 145 Questions  

To operationalize the foundational components, a detailed evaluation framework comprising 145 questions across 10 critical categories was designed, of which 25% is covered in an industry-standard model card that serves as a minimal baseline for any healthcare AI vendor.

  1. Model or System Information 
  2. Model or System Use Case & Application  
  3. Benefits & Expected Impact   
  4. Detailed System Information   
  5. Risks & Limitations (Warnings)  
  6. Detailed AI/ML Information  
  7. Key Metrics  
  8. Transparency Information  
  9. Privacy & Security  
  10. Implementation & Pricing  

Each category was mapped to one or more foundational components, creating a holistic and consistent evaluation process.   

 A color-coded scoring rubric then enabled MSMC to make evidence-based decisions quickly and transparently. Vendor responses were reviewed against predefined thresholds and summarized visually for MSMC’s governance committee.  

Results

By applying the AI Risk Evaluation Framework to one of its key AI vendors, MSMC was able to surface critical insights that informed both procurement and clinical governance decisions.   

The evaluation produced several quantifiable and strategic outcomes: 

Improved Accountability:

MSMC strengthened “human-in-the-loop” controls to maintain clinical oversight in AI-assisted workflows.   

Model Transparency:

Their AI vendor was prompted to share detailed performance metrics, giving MSMC clear visibility into accuracy, monitoring, and failure modes.   

Data Governance Awareness:

MSMC identified and addressed misalignments in data-sharing practices, leading to more informed vendor negotiations.  

Safety and Feedback Integration:

The evaluation emphasized the importance of internal safety escalation protocols and collaborative feedback loops with its AI vendor.  

a pair of glasses on a notebook

Long-Term Benefits

  • Informed Vendor Selection: The framework now serves as a repeatable tool for evaluating future AI vendors, reducing risk at the point of procurement.  
  • Cross-Functional Alignment: The process fostered collaboration between clinical, compliance, and IT teams, embedding AI risk awareness across the organization.  
  • Foundation for Continuous Monitoring: The evaluation surfaced insights to prepare MSMC for ongoing model performance tracking and governance in partnership with their vendors, supporting long-term patient safety and regulatory readiness.  

Mount Sinai’s proactive approach to AI governance positions the organization as a leader in responsible healthcare innovation. By building a repeatable, transparent evaluation framework, MSMC not only strengthened its own clinical and operational safeguards but also set a new standard for how healthcare systems can adopt AI confidently and ethically with patient safety and trust at the center.  

Contact Us

Let's Start a Conversation Today!