Associate Data Engineer
PayU is hiring an Associate Data Engineer in Gurgaon to design, build, and operate near-real-time data pipelines and lakehouse platforms on AWS. Day to day: developing Flink/Kafka pipelines, managing Iceberg data lakes, optimizing queries with Spark, Trino, and StarRocks, and ensuring data quality and reliability.
- Compensation
- Not disclosed
- Location
- Gurgaon, IN
- Work setup
- Not specified
- Seniority
- Junior
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 2–4 years of professional experience in Data Engineering, Software Engineering, Analytics Engineering, or related discipline
- Hands-on experience in Python and SQL
- Good understanding of data structures, algorithms, relational databases, and software development principles
- Understanding of distributed systems, data processing, and large-scale data architectures
- Experience building, maintaining, or supporting ETL and ELT data pipelines
- Experience working with data warehouses, data lakes, or lakehouse platforms
- Understanding of data modeling, dimensional modeling, and DataMart development
- Experience with batch processing; exposure to streaming/event-driven pipelines is an advantage
- Understanding of SQL query optimization, data partitioning, indexing, and performance tuning
- Experience with relational databases and familiarity with NoSQL databases
- Hands-on experience with cloud platforms, preferably AWS (S3, EMR, Redshift, MSK)
- Practical experience with some of: Spark/PySpark, Airflow, Kafka, Debezium, Flink, Iceberg, Trino, StarRocks, ClickHouse, Hadoop
- Experience with Git, code reviews, testing, and CI/CD practices
- Kafka/Kafka Connect, Debezium/Maxwell
- Binlog/WAL/GTID log replication
- Ability to troubleshoot issues, analyze problems, and communicate solutions clearly
- Strong communication and collaboration skills
Nice to have
- Experience with Change Data Capture, database replication, or incremental data processing
- Familiarity with data quality frameworks, pipeline monitoring, and observability practices
- Exposure to Docker and Kubernetes
- Familiarity with infrastructure automation and DevOps practices
- Experience with on-premises or hybrid data platforms
- Exposure to Apache Iceberg, lakehouse architecture, or open table formats
- Experience in fintech, payments, banking, or other high-volume transaction environments
- Familiarity with NoSQL databases such as Cassandra, MongoDB, or Redis
- Understanding of Agile development methodologies and engineering best practices
Full description
Associate Data Engineer / Data Engineer
Experience: 2–4 Years
About the Team
Our Data Engineering team is the backbone of PayU’s data-driven organization. We design, build, and operate scalable data platforms that support critical business decisions across multiple domains.
Working with large and diverse data volumes, the team enables reliable data movement from operational systems to data warehouses, data lakes, lakehouses, and analytics platforms. We collaborate closely with business and technology stakeholders to build fit-for-purpose data solutions that address evolving organizational needs.
About the Role
As an Associate Data Engineer / Data Engineer, you will contribute to the design, development, and maintenance of scalable data pipelines and processing solutions. You will work with batch and real-time data platforms, support data warehouse and lakehouse initiatives, and help ensure the performance, quality, reliability, and accessibility of enterprise data.
This role provides an opportunity to work with modern cloud and open-source technologies while developing expertise across data warehousing, data lakes, lakehouses, and analytical serving platforms.
Key Responsibilities
- Work with data engineers, architects, product teams, and business stakeholders to understand data requirements and deliver reliable technical solutions.
- Participate in requirements gathering, solution design, development, testing, deployment, and support for data engineering initiatives.
- Design, develop, and maintain reliable near-real-time (NRT) data pipelines using technologies such as Apache Kafka, Apache Flink, Python, and SQL.
- Build and support NRT data processing workflows that enable timely analytics, monitoring, reporting, and downstream business use cases.
- Develop and maintain Flink-based data lake solutions for processing high-volume, continuously changing data.
- Work with Apache Iceberg as the table and data format for building scalable, reliable, and queryable data lakehouse platforms.
- Contribute to data ingestion, transformation, schema evolution, partitioning, compaction, and data lifecycle management within Iceberg-based data lakes.
- Develop and maintain analytical datasets, data models, and DataMart solutions for reporting and business intelligence.
- Build and optimize data processing and query workloads using open-source technologies such as Apache Spark, Trino, and StarRocks.
- Support low-latency analytical serving and interactive query use cases using open-source query engines such as .