Senior Data Engineer – Python, BigQuery and DBT

Overview:

We are looking for a Senior Data Engineer – Python, BigQuery and DBT to join our Index Engineering team in Gurgaon. You will be part of a team that builds and evolves critical data platforms on a modern cloud-native stack using dbt, BigQuery, and Apache Airflow on Google Cloud Platform. You will work with large-scale financial datasets, including securities master data, corporate actions, and market data from external vendors — designing robust ELT pipelines, improving data models, and driving platform enhancements using modern engineering practices. This is a hands-on role for someone with strong data engineering fundamentals — someone who understands how to design scalable pipelines, model data effectively, and build reliable systems that serve business-critical workloads.

Responsibilities:

Data Modelling & Transformation

• Design and build data models that accurately represent business domains — applying dimensional modelling, slowly changing dimensions, and normalisation/denormalisation trade-offs appropriate to the use case.

• Develop and maintain data transformation logic using dbt on BigQuery — leveraging models, macros, incremental strategies, and tests to keep transformations modular, version-controlled, and well-documented.

• Define and enforce naming conventions, modelling standards, and layering practices (staging, intermediate, marts) across the data warehouse.

Data Pipeline Engineering

• Design, build, and maintain ELT pipelines that ingest, transform, validate, and serve financial data at scale — with a focus on reliability, idempotency, and observability.

• Build ingestion frameworks for external data vendor feeds — handling diverse file formats, schema variations, validation rules, reconciliation, and error recovery.

• Orchestrate pipeline workflows using Apache Airflow (Cloud Composer), managing dependencies, retries, SLAs, and alerting.

• Design and implement full- and incremental-load strategies, backfill mechanisms, and pipeline-recovery patterns.

Data Quality & Reconciliation

• Implement data reconciliation processes to verify accuracy across upstream sources and internal datasets — building automated checks for row counts, value matches, and business rule compliance.

• Define and enforce data quality standards through automated testing, validation layers, and monitoring — treating data quality as a first-class engineering concern.

• Set up monitoring and alerting for pipeline health and data freshness using tools such as Datadog.

Platform & Performance

• Design partitioning, clustering, materialisation, and caching strategies to optimise query performance and manage storage costs in BigQuery.