Senior Data Engineer - Databricks & Streaming - Healthcare AI (Onsite, Evening Shift, Lahore, PKR Salary)

Requirements:

• 4+ years of experience in data engineering, with substantial production experience in Databricks.
• Strong experience with Spark SQL, PySpark, Delta Lake, Medallion Architecture, and Delta Live Tables (DLT).
• Hands-on experience with Structured Streaming or equivalent production-grade streaming ingestion using Azure Event Hubs, Kafka, or Kinesis.
• Strong understanding of checkpoint recovery, watermarking, and deduplication strategies.
• Demonstrated experience debugging source-to-warehouse data discrepancies.
• Ability to walk through a real-world incident involving mismatched record counts and explain how the root cause was identified and resolved.
• Proven experience integrating third-party REST APIs in production.
• Experience handling pagination edge cases, rate and row limits, retries, and schema drift.
• Experience with entity resolution or data matching involving messy, real-world text data.
• Experience with a metrics or semantic layer such as Holistics AML/AQL, dbt Metrics, or LookML.
• Working understanding of why non-additive measures cannot be reliably calculated from pre-aggregated rollups.
• Strong SQL and Python skills, with the ability to own data pipelines end-to-end with minimal oversight.
• Strong written and spoken English, with the ability to collaborate effectively with a US-based team asynchronously.
• Healthcare data experience, including referrals, payer taxonomy, claims/eligibility, or other PHI-adjacent datasets.
• Familiarity with HIPAA handling expectations.
• Experience with voice-agent, call-center, or telephony/conversation data.
• Familiarity with call transcripts and containment or outcome metrics.
• Hands-on experience with Holistics, specifically AML/AQL modeling.
• Experience with the broader Azure ecosystem beyond Event Hubs, including ADLS, ADF, and Key Vault.

Responsibilities:

• Build and maintain resilient ingestion pipelines for third-party vendor REST APIs.
• Work primarily with voice-AI observability and telephony platforms.
• Handle different pagination schemes, including offset/limit and page/cursor models.
• Manage row and rate limits through time-windowing and adaptive bisection.
• Implement robust schema-drift handling through contract and column-presence checks.
• Ensure alerts are triggered when fields are renamed, moved, or removed rather than silently propagating null values.
• Own streaming ingestion from Azure Event Hubs into Databricks using Structured Streaming and/or Auto Loader.
• Manage checkpoints and offsets, watermarking, and at-least-once deduplication.
• Perform source-parity reconciliation across ingested and production data.