Info Services
Senior Data Engineer/AI Data Platform Engineer - FinOps
- Compensation
- Not disclosed
- Location
- Hyderabad
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 7+ years
- Category
- Data engineering
- Posted
Skills
Full description
About the Role :
We are looking for a highly skilled Data Engineer to join our team. You will play a key role in building scalable data pipelines, enabling efficient data landing into Delta Lake, and driving insights through modern data platforms like Databricks, AWS, Snowflake, BI tools, and emerging GenAI technologies.
This role is ideal for someone who enjoys solving complex data challenges, optimizing performance, and working in a fast-paced, startup-style environment.
Key Responsibilities :
- Design and implement robust data pipelines for ingestion and transformation.
- Perform data landing into Delta Lake with proper partitioning, schema evolution, and optimization.
- Design and implement Medallion Architecture (Bronze, Silver, Gold layers).
- Develop and manage data workflows using Databricks (PySpark-based).
- Build and maintain data pipelines orchestrated via Apache Airflow.
- Work with AWS services (S3, Redshift, EMR, Glue, Lambda) for data ingestion, storage, and processing.
- Work with Snowflake for data warehousing, including :
- Managing virtual warehouses and workload optimization
- Analyzing query history and performance tuning
- Leveraging information schema & account usage/stats schema
- Working with AWS CUR (Cost & Usage Report) logs for cost analysis and optimization
- Build scalable Lakehouse architectures for analytics and reporting.
- Optimize data processing jobs for performance, cost, and reliability.
- Collaborate with analytics teams and support BI tools (Looker, Tableau, Power BI).
- Develop and maintain LookML models for business reporting.
- Implement data governance, cataloging, and security best practices.
- Support and enable GenAI and ML use cases by building curated, high-quality datasets.
- Work with teams to integrate LLM-powered solutions into data workflows (where applicable).
- Monitor pipelines using observability tools and ensure high availability.
- Troubleshoot production issues and continuously improve system performance.
Required Skills & Experience :
- Bachelors degree in Computer Science or equivalent practical experience
- 7+ years of strong experience in Data Engineering.
- Advanced proficiency in Python and PySpark.
- Hands-on experience with Databricks platform.