AI Quality and Evaluation Lead

Job Description

AI Quality and Evaluation Lead:
On behalf of our financial services client, Procom is searching for an AI Quality and Evaluation Lead for a permanent role. This position is a hybrid position with 3 days onsite at our client’s Toronto office.

AI Quality and Evaluation Lead - Job Description:
The AI Quality and Evaluation Lead will be responsible for developing and implementing evaluation strategies and quality standards for AI solutions within the organization. This role involves working closely with various stakeholders to ensure AI-enabled mortgage workflows meet business policy, risk appetite, and user expectations through measurable acceptance criteria.

AI Quality and Evaluation Lead - Responsibilities:
• Own and continuously evolve the AI Factory quality and evaluation strategy
• Define use case specific quality measures with cross-functional teams
• Establish quality and evaluation gates within production readiness standards
• Conduct failure mode testing in collaboration with Information Security
• Monitor production quality and define triggers for investigation or redesign
• Create defect taxonomies and root cause practices
• Evolve the function to a federated operating model with reusable scorecards

AI Quality and Evaluation Lead - Mandatory Skills:
• Bachelor’s degree in computer science, engineering, data science, or related discipline
• 7+ years of experience in quality engineering, software testing, or AI governance
• Experience designing evaluation frameworks for production AI systems
• Strong skills in Python, SQL, test automation, and evaluation tooling
• Knowledge of accuracy metrics, experiment design, and risk controls
• Ability to translate business policy into measurable acceptance criteria
• Experience in financial services or regulated technology environments

AI Quality and Evaluation Lead – Nice-to-Have Skills:
• Experience with RAG, document intelligence, and agent tools
• Familiarity with MLflow, Ragas, DeepEval, or comparable frameworks
• Exposure to model risk, credit risk, or independent validation environments
• Experience in mortgage lending or lending operations
• Understanding of third-party partnerships in financial services

AI Quality and Evaluation Lead – Assignment Length:
This is a permanent position.

AI Quality and Evaluation Lead - Start Date:
ASAP.

AI Quality and Evaluation Lead - Assignment Location:
Toronto, Ontario, Canada. This is a hybrid position requiring 3 days in the office.

This employer uses both human and technology-assisted tools to support candidate screening and assessment. Final hiring decisions are made by people.