Sr Data Engineer - Mississauga
Senior data engineer on a 6-month, 100% remote contract via Randstad, building high-volume ingestion pipelines in Databricks Delta Lake (Bronze/Silver/Gold) and porting SQL Server stored procedures to PySpark/Spark SQL for a healthcare pharmacy compliance platform, with emphasis on data quality, testing, and legacy-parity reconciliation.
- Compensation
- Not disclosed
- Location
- Mississauga, Canada
- Work setup
- Remote
- Seniority
- Senior
- Employment
- Contract
- Experience
- 3+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 6+ years of dedicated professional data engineering experience
- 3+ years of hands-on production experience with Databricks and Apache Spark
- Strong skills independently writing, testing, and tuning production-grade Python and PySpark
- Advanced SQL and T-SQL proficiency, including translating complex SQL Server stored procedures
- Practical Delta Lake experience (MERGE/upsert, incremental loads, schema evolution, time travel, table maintenance)
- Deep understanding of Spark execution model and performance troubleshooting
- Proficiency with Git, automated data testing frameworks, and CI/CD tools
- Strong written communication and technical documentation skills
Nice to have
- Experience with healthcare, pharmacy, payer, or life-sciences data (HL7, ePrescribe, NCPDP, claims, NDC, HIPAA PHI)
- Direct experience with SQL Server to Databricks/Lakehouse migrations
- Familiarity with Azure Data Factory, Azure Storage, Delta Live Tables, Databricks Workflows, and Unity Catalog
- Exposure to data quality tools like Great Expectations or transformation frameworks like dbt
- Active Databricks Certified Data Engineer (Associate or Professional) designation
Full description
Our client, is seeking an experienced, detail-oriented Sr Data Engineer to join their platform modernization team.
In this technical, hands-on role, you will build pipelines, transformations, and data validation tools to support the modernization of a high-stakes healthcare pharmacy compliance platform using Databricks. Working within an Agile delivery pod under a technical lead, you will engineer high-volume ingestion workloads across a multi-tier Delta Lake architecture (Bronze, Silver, Gold) while porting and re-implementing existing SQL Server stored procedure logic into PySpark and Spark SQL. Because this platform drives financial and regulatory decisions, engineering efforts prioritize data accuracy, automated testing, parity reconciliation against legacy outputs, and full traceability across every pipeline step.
Duration: 6-Month Contract (100% Remote)
Work Arrangement: 100% Remote
Advantages
100% Remote Flexibility: Enjoy complete remote work flexibility while contributing to a mission-critical platform modernization initiative.
Modern Lakehouse Architecture: Hands-on technical execution utilizing Databricks, PySpark, Spark SQL, and Delta Lake design patterns.
High-Impact Regulatory Domain: Drive data pipelines for a platform handling critical healthcare and pharmacy compliance workflows where precision and data quality are paramount.
Focused 6-Month Engagement: High-intensity consulting assignment centered on core legacy-to-cloud platform migration and reconciliation.
Responsibilities
Delta Lake Pipeline Engineering: Build and maintain high-volume batch and intraday ingestion pipelines into a Delta Lake architecture (Bronze, Silver, Gold layers).
Stored Procedure Migration: Port, translate, and re-implement legacy SQL Server stored procedure business logic into optimized PySpark and Spark SQL code.
Data Quality & Reconciliation: Develop automated test suites, data quality checks, and output-parity reconciliation jobs comparing new lakehouse results against legacy system outputs.
Performance Tuning & Optimization: Optimize Spark job execution by tuning partition strategies, file sizing, join types, skew handling, and cluster compute parameters.
Idempotency & Late Data Handling: Construct idempotent, re-runnable data pipelines designed to gracefully manage late-arriving or corrected data updates.
CI/CD & Engineering Standards: Maintain version control in Git, participate in peer code reviews, build CI/CD pipeline automation, and document technical design specs.
Qualifications
Data Engineering Depth: 6+ years of dedicated professional experience in data engineering, including 3+ years of hands-on production experience with Databricks and Apache Spark.