Senior Data Engineer

Role Overview :

The ideal candidate will have strong hands-on experience with Python, SQL, Spark, ETL, cloud technologies, and data lake environments. You will work with data engineers, developers, analysts, and other technical teams to make data easily accessible, reliable, and optimized for analytics.

Key Responsibilities :

• Develop and maintain scalable data pipelines for ingesting data from multiple sources.

• Build data transformation and processing workflows using Spark, Python, and SQL.

• Work with data lake environments and distributed data processing frameworks.

• Integrate data from different source systems into centralized analytical platforms.

• Develop ETL workflows to support reporting, analytics, and business applications.

• Design and maintain data models for analytical and reporting requirements.

• Write optimized SQL queries for data extraction, transformation, and analysis.

• Work with technologies such as Spark, Hadoop, YARN, and distributed file systems.

• Support cloud-based data engineering environments across AWS or GCP.

• Develop scripts using Python and Shell to automate recurring data engineering and operational tasks.

• Monitor data pipelines and platform services to identify failures and performance issues.

• Investigate and resolve data pipeline, infrastructure, and application-related issues.

• Optimize data processing jobs and SQL queries to improve performance.

• Support development, testing, and production data environments.

• Work closely with software developers, database architects, data analysts, and business teams.

• Assist in developing reporting datasets, dashboards, KPIs, and analytical solutions.

• Contribute to automated reporting and recurring analytical workflows.

• Maintain technical documentation for data pipelines, data models, and operational processes.

• Participate in Agile development processes and software deployment activities.

• Provide production and on-call support as required.

Technical Requirements :

• 5+ years of experience in Data Engineering, Big Data, or related software engineering roles.

• Strong hands-on experience with Apache Spark and distributed data processing.

• Good experience working with data lakes, Hadoop, YARN, or distributed file systems.

• Strong SQL skills and experience working with large datasets.

• Experience with Python and Shell scripting.

• Hands-on experience developing ETL pipelines and data transformation workflows.

• Good understanding of OLAP concepts and analytical data processing.

• Experience with AWS or GCP cloud services.

• Exposure to AWS EMR, Hadoop, or similar big data platforms is beneficial.

• Experience with Vertica, Dremio, or other big data SQL technologies is an advantage.

• Understanding of relational databases and data structures.

• Familiarity with Git or other version-control systems.

• Experience working in Linux-based environments.

• Exposure to visualization tools such as Tableau or Apache Superset is a plus.

• Understanding of data monitoring, debugging, and troubleshooting practices.

Required Skills & Competencies :

• Strong data analysis and data-wrangling capabilities.

• Good understanding of scalable data pipeline architecture.

• Strong analytical and problem-solving skills.

• Ability to troubleshoot complex data and production issues.

• Ability to work independently while collaborating effectively with cross-functional teams.

• Good communication and documentation skills.

• Comfortable working in an Agile environment.

• Willingness to participate in on-call support when required.

Education & Experience :

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

• 4+ years of professional experience in software or data engineering.