Data Engineer

Data Engineer/DeveloperRole Summary

Design, build, and optimize Azure data pipelines and lakehouse solutions.
Deliver secure, reliable datasets with strong governance, automation, and
documentation. Collaborate across teams and contribute to standards in an Agile
setting.

Must-Have (Day 1)

• Experience:

4–6 years in data engineering

• Core

Platform: Databricks with Python, Spark, Pandas (notebooks and modular
code)

• Orchestration:

Azure Data Factory (pipelines, integration runtimes); ingest from diverse
sources

• Lakehouse:

Delta Lake fundamentals; Medallion architecture (bronze/silver/gold) in
production

• Storage/SQL/Performance:

Azure Data Lake Storage (ADLS); strong SQL; performance-aware design

• Data

Patterns: ETL/ELT; data modeling (e.g., dimensional/star schema)

• DevOps

& Security: CI/CD for data projects (Azure DevOps or GitHub
Enterprise); familiarity with Azure Entra ID for SSO/RBAC; secure
workspace/data access

• Quality

& Observability: Data validation/testing, code reviews, and basic
monitoring/alerting for jobs/pipelines

• Ways of

Working: Agile/Scrum (Jira/Confluence); clear pipeline and data contract
documentation

• Collaboration:

Effective stakeholder engagement; support/mentor junior team members;
clear communication

• Generative

AI (Day 1):

• Prompt

design for data tasks (ingestion, transformations, documentation) with
clear objectives and constraints

• Use of

Copilot/ChatGPT to scaffold notebooks/jobs, generate tests, and optimize
SQL/Spark—validates outputs before merging

Nice-to-Have (Train within 60–90 days)

• Unity

Catalog migration (Hive to Unity) and permissions/governance

• Databricks

DevOps (cluster configuration, secret management, workspace automation)

• Azure