Senior Data Engineer

Job Summary

• We are seeking a Senior Data Engineer to join our Data & Analytics team. The ideal candidate will design, build, and maintain robust on-premise data pipelines, architect flexible lakehouse solutions, and lead data infrastructure initiatives to enable data-driven decision-making across the organization.

Mandatory Skill-set

• Bachelor's degree in Computer Science, IT, Engineering, or related field.
• Must have 5 years of hands-on experience in data engineering and data pipeline development;
• Hands on experience in SQL, with in-depth expertise in Microsoft SQL Server, including query optimization, indexing strategies, performance tuning, and troubleshooting;
• Hands-on experience with at least one NoSQL database, such as MongoDB or Apache Cassandra;
• Practical experience in distributed data processing using Apache Spark and PySpark, including performance optimization and large-scale data processing;
• Strong Python programming skills for data processing, automation, and developing scalable, reliable, production-grade data pipelines;
• Expertise in designing, deploying, and managing containerized data pipelines using Kubernetes in on-premises environments;
• Strong understanding of relational and non-relational data modeling concepts, including schema design and optimization;
• Hands-on experience with Apache Airflow or similar orchestration tools for scheduling, monitoring, and managing batch and real-time data pipelines;
• Knowledge of data security and governance practices, including role-based access control (RBAC), data masking, and compliance with relevant regulatory and organizational standards;

Desired Skill-set

Good to have experience with on-premise data virtualization or logical data warehouse concepts.

Good to have real-time streaming technologies (Kafka, Apache Flink);

Responsibilities

• Design and develop autonomous, production-grade ETL/ELT data pipelines using Python and PySpark that ingest, transform, and deliver high-quality data while maintaining integrity and performance standards.
• Implement and manage flexible lakehouse architecture across raw, curated, and consumption layers, including data partitioning, cataloging, and metadata management;
• Deploy and manage data pipelines using Kubernetes and Docker to ensure scalability, reliability, and efficient resource utilization in on-premise environments;
• Leverage strong SQL Server expertise to design optimal data models, write complex queries, and perform query optimization across the data platform;
• Establish and maintain robust CI/CD practices for data pipeline deployment, including automated testing, version control, and continuous monitoring;
• Enforce security, governance, and role-based access controls across all data layers while ensuring compliance and auditability;