Data Engineer II
Data Engineer II at DAT building and maintaining cloud-native ELT data pipelines in Snowflake using dbt, Apache Airflow, Python, and GitHub Actions, with data quality testing and AI-assisted development. Onsite in Bengaluru with work hours 1:00 PM to 10:00 PM IST.
- Compensation
- Not disclosed
- Location
- Bangalore, Karnataka, India
- Work setup
- Not specified
- Seniority
- Middle
- Employment
- Internship
- Experience
- 3+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 3-5 years of professional experience in Data Engineering, Analytics Engineering, or similar technical role (including relevant internship experience)
- Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related quantitative field
- Strong proficiency in Python
- Hands-on experience with a cloud data warehouse, preferably Snowflake
- Basic experience creating or running jobs/workflows using an orchestration tool like Apache Airflow
- Experience with Git and CI/CD pipelines, ideally GitHub Actions
- Comfort using AI-assisted coding tools in a professional engineering workflow
- Strong problem-solving abilities, excellent attention to detail, and a proactive, collaborative approach to teamwork
Nice to have
- Familiarity with data transformation concepts and tools, dbt a bonus
Full description
About DATDAT is an award-winning employer of choice and a next-generation SaaS technology company that has been at the leading edge of innovation in transportation supply chain logistics for 45 years. ders optimistically share future possibilities to inspire and motivate others toward their full potential. We expect our employees to find ways to embrace positive change, be curious and challenge the status quo, and provide solutions to unmet problems. Joining DAT means joining a culture focused on fostering development, building genuine connections, recognizing each other’s strengths and sharing in successes.
We continue to transform the industry year over year, by deploying a suite of software solutions to millions of customers every day - customers who depend on DAT for the most relevant data and most accurate insights to help them make smarter business decisions and run their companies more profitably. We operate the largest marketplace of its kind in North America, with 400 million freights posted in 2022, and a database of $150 billion of annual global shipment market transaction data. Our headquarters are in Denver, CO, with additional offices in Missouri, Oregon, and Bangalore, India. For additional information, see.
Key Technologies
- Data Warehouse: Snowflake/Data Lake
- Transformation: dbt (data build tool)
- Orchestration: Apache Airflow
- CI/CD: Git, GitHub Actions
- Programming: Python
- AI-Assisted Development: LLM-based coding assistants (e.g., Claude, Copilot) for code generation, review, and documentation
Core ResponsibilitiesData Pipeline Development & Engineering
- ELT Process Implementation: Assist senior engineers in designing, building, testing, and maintaining cloud-native ELT (Extract, Load, Transform) data pipelines, ensuring data is reliably loaded into Snowflake and S3 Iceberg layer.
- Pipeline Architecture: Contribute to the design of scalable, modular data pipelines that support incremental loads, backfills, and reprocessing with minimal manual intervention.
- Transformation with dbt: Develop and maintain data models using dbt (data build tool) for data cleaning, aggregation, and transformation and utilize the Snowflake data warehouse.
- Python Scripting: Utilize Python to build custom data extraction scripts, implement monitoring tools, and contribute to general automation efforts.
- Pipeline Reliability: Build in retry logic, alerting, and failure-handling patterns so pipelines degrade gracefully and self-recover where possible.
Workflow Orchestration and Automation
- Airflow DAGs: Learn to author, schedule, and monitor data workflows defined as Directed Acyclic Graphs (DAGs) in Apache Airflow.