Lead Machine Learning Engineer - MLOps

About the Job :

We are looking for a Lead Machine Learning Engineer to improve the reliability of Tonik's ML deployment and operations. You will remain a hands-on senior individual contributor, own MLOps project or topic areas, and review others' work.

Your priority is to reduce deployment mistakes and establish engineering practices that make frequent scorecard safe and repeatable. You will support real-time and batch inference, including underwriting and offer-generation workflows, while building reusable controls that can be adopted across the ML estate.

Job Responsibilities :

• Assess the current ML delivery process, identify recurring failure modes and own a practical MLOps improvement roadmap. Personally implement priority changes and demonstrate their effect on release quality, recovery time and delivery effort.

• Design and build reliable real-time and batch inference workflows using Python, SQL, APIs and cloud services. Work within Tonik's GCP and BigQuery environment and API-based model serving architecture, including FastAPI, Docker and Kubernetes.

• Introduce repeatable release controls : versioned model artifacts, feature schemas and configuration; separate environments; automated tests; peer review; approval evidence; deployment checks; and rollback procedures.

• Prevent training-serving inconsistencies through explicit feature contracts, dependency pinning and version alignment. Reconcile production scores against approved reference outputs, including missing-data handling, score ranges, stacked models and segment routing.

• Build CI/CD pipelines for ML that test feature transformations, inference logic, APIs and integrations. Add regression datasets, schema checks and release validation so changes cannot silently alter approved scoring behaviour.

• Establish model registries, experiment and artifact traceability, reproducible packaging and auditable release records. Automate retraining and evaluation where useful, with explicit review and approval before production promotion.

• Improve observability for data freshness, feature availability, scoring failures, latency, batch completion and model versions. Implement actionable alerts, incident runbooks and post-release checks with clear ownership.

• Partner with DS, Risk, IT and Product on safe scorecard changes, including component-score replacements, offer-system integrations and third-party data changes. Support 3 - 6 month refresh cycles and urgent changes with appropriate testing.

• Review others' work, including architecture, code, test coverage and deployment evidence. Mentor through hands-on technical guidance, lead project or topic areas, and introduce proven industry practices suited to Tonik's scale.

• Work with IT and security teams on least-privilege access, secrets, sensitive data handling, resilience and production support. Balance reliability, inference performance and infrastructure cost.

• Uphold model governance for ML systems : reproducibility, explainability and regulatory and compliance requirements in a banking environment.

• Build reusable ML tooling (experiment tracking, model validation pipelines, retraining jobs and inference utilities) that Data Scientists and MLEs adopt across the function.

Required Experience and Qualifications :

• At least 5 years of core MLE experience building, deploying and operating production ML systems. The experience should include direct ownership of inference services, release automation and production reliability.

• A bachelor's or master's degree in computer science, software engineering, engineering or a related discipline.

• Strong Python, SQL and software engineering skills, including modular design, testing, Git, code reviews and dependency management. Practical understanding of model preprocessing and inference.

• Hands-on experience with Docker, Kubernetes, REST APIs and a Python serving framework such as FastAPI; experience supporting both real-time and scheduled batch workloads.

• Demonstrated implementation of CI/CD for ML, artifact and model versioning, automated validation, environment promotion, monitoring and rollback.

• Production experience on GCP, AWS or Azure, with the ability to diagnose failures across data, application and infrastructure layers.

• Evidence of improving release quality or resolving recurring deployment defects, and the ability to lead technical projects and review others' work as a senior individual contributor.

Preferred Experience and Qualifications :

• Experience in banking, fintech or other environments with auditable model approvals and strict production change controls; familiarity with credit scorecards and stacked-model inference.

• Experience with GCP, BigQuery, FastAPI and Kubernetes; tools such as MLflow or equivalent registries and workflow orchestration platforms.

• Experience with feature stores, training-serving consistency, infrastructure as code, staged rollouts or shadow scoring.

• Experience defining service reliability measures, running incident reviews and improving production support practices.

What Success Looks Like :

• Recurring deployment errors decline, and model releases have automated checks, traceable evidence and tested rollback paths.

• Score and feature reconciliation catches mismatches before production; monitoring makes failures visible and actionable.

• Reusable MLOps components support faster refreshes, and technical reviews improve engineering consistency.

Who are we looking for?

We value teamwork, honesty, reliability, practical problem-solving and a sense of humour. You take ownership of your work, communicate early when something is wrong, and challenge assumptions respectfully. You are comfortable collaborating across locations and improving how the team works.

• Willingness to go the extra mile for success.

• Energetic and lively, and street smart.

• As our approach to banking is far from ordinary, we are looking for an innovative and creative individual to join our team.

• Above all, a great team player and proactive communicator. This is a table-stake criterion.