Sr Big Data Engineer
Senior Data Engineer at Capco (a Wipro consulting firm) in Pune, building and optimizing large-scale ETL/ELT data pipelines on AWS and Databricks using Python, SQL, and PySpark for banking and financial-services clients. Requires 5+ years of experience and includes a mandatory in-person interview at the client's Pune office.
- Compensation
- Not disclosed
- Location
- India - Pune
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 10+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 5+ years of relevant Data Engineering experience
- Strong hands-on Python for production-grade data engineering applications
- Expert-level SQL including complex joins, window functions, and CTEs
- Data modeling, query optimization, and performance tuning
- Hands-on experience with PySpark and Apache Spark distributed data processing
- Hands-on experience with AWS cloud-based data engineering solutions
- Experience working with Databricks for large-scale data processing
- Experience designing and optimizing ETL/ELT pipelines
- Experience integrating data from multiple structured and semi-structured sources
- Experience implementing CI/CD pipelines and DevOps best practices for data workloads
- Understanding of version control, automated testing, and deployment automation
- Knowledge of containerization, orchestration, and cloud-native architectures
- Understanding of data governance, data quality frameworks, and data security
- Experience with data validation, monitoring, and compliance controls
- Proven experience with large-scale data platforms and distributed processing frameworks
- Strong analytical, debugging, problem-solving, and communication skills
- Experience working in Agile/Scrum environments
- Bachelor's or Master's degree in Computer Science or Information (Technology)
- Ability to collaborate with cross-functional technical and business stakeholders
- Mandatory in-person interview round at the client's office in Pune
Nice to have
- Exposure to AWS services such as S3, Glue, Lambda, EMR, and Redshift
- Experience with Azure or GCP cloud platforms
- Familiarity with machine learning concepts, MLOps practices, and AI/ML-enabled data platforms
- Exposure to Generative AI, LLMs, RAG, and AI agents
- Knowledge of Docker, Kubernetes, and cloud-native deployment patterns
- Experience with modern data lakehouse architectures and advanced Databricks capabilities
Full description
Job Title: Big Data EngineersAbout Us
“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery.
WHY JOIN CAPCO?
You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry.
MAKE AN IMPACT
Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.
#BEYOURSELFATWORK
Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.
CAREER ADVANCEMENT
With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.
DIVERSITY & INCLUSION
We believe that diversity of people and perspective gives us a competitive advantage.
Position: Data Engineer
Location: Pune, Maharashtra
Job Description
The Data semantic solution team is looking for a Big Data Engineer to drive our mission to unlock potential of data assets by consistently innovating, eliminating friction in how users access data from its Big Data repositories and enforce standards and principles in the Big Data space. The candidate will be part of an exciting, fast paced environment developing Data Engineering solutions in the data and analytics domain.
Role
- Develop high quality, secure and scalable data pipelines using spark, Scala/ python on Hadoop or object storage. • Leverage new technologies and approaches to innovate with increasingly large data sets. • Drive automation and efficiency in Data ingestion, data movement and data access workflows by innovation and collaboration. • Understand, implement and enforce Software development standards and engineering principles in the Big Data space. • Work closely with business stakeholders and embedded engineering teams within business teams in a collaborative manner to help them build scalable products in quick time. • Contribute ideas to help ensure that required standards and processes are in place and actively look for opportunities to enhance standards and improve process efficiency. • Perform assigned tasks and production incident independently.