Data Engineer - Databricks Pipeline Engineering & Architecture

About the Role :

We are seeking an analytical and highly skilled Data Engineer with a strong background in Databricks and deep domain experience in Manufacturing or Supply Chain Planning. In this role, you will be part of our software engineering division, designing and executing data-driven solutions for global enterprise clients.

Your primary focus will be transforming complex physical operations (factory constraints, logistics, and inventory) into highly optimized data models. These models will directly power business-critical systems, including simulation engines, capacity planning algorithms, and risk analysis tools.

Role & Responsibilities : 1. Databricks Pipeline Engineering & Architecture :

• Design, develop, and maintain high-volume ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).

• Implement Medallion Architecture (Bronze, Silver, Gold layers) to process both real-time IoT and historical enterprise data.

• Optimize Databricks cluster utilization and query performance to ensure compute resources are running cost-effectively.

• Enforce data governance, validation rules, and rigorous data quality checking at every stage of the pipeline.

2. Analytical & Simulation Modeling Support :

• Structure and expose highly curated data layers optimized for deterministic or stochastic simulation workloads (e.g., Monte Carlo simulations, "What-If" operational scenarios).

• Collaborate with Data Scientists and Operations Research teams to build data foundations for capacity planning, factory throughput analysis, and resource optimization engines.

• Ingest, map, and process supply chain risk variables (vendor delay probabilities, weather patterns, material shortages) into structured data sets for risk analysis platforms.

3. Supply Chain Domain Data Integration :

• Map and model intricate manufacturing and logistics data structures, including Bill of Materials (BOM), production routing, lead times, safety stock, and demand forecasts.

• Integrate and unify data from disparate legacy enterprise systems such as ERP (SAP, Oracle), MES (Manufacturing Execution Systems), and WMS (Warehouse Management Systems).

Required Skills & Qualifications : Technical Capabilities :

• Databricks Ecosystem : 3+ years of dedicated hands-on experience using Databricks, PySpark, Spark SQL, and Delta Lake.

• Programming : Advanced proficiency in Python or Scala, and expert-level SQL for writing highly optimized, complex analytical queries.

• Data Modeling : Proven experience designing data models specifically suited for time-series operational data, manufacturing constraints, or multi-echelon supply chain networks.