Big Data Architect - Python/ETL

Role & Responsibilities :

• Design and architect scalable, secure, and high-performance big data platforms capable of handling large volumes of structured and unstructured data.

• Define end-to-end data architecture, data ingestion, processing, storage, integration, and analytics solutions.

• Develop and implement data pipelines using technologies such as Apache Spark, Kafka, Hadoop, Hive, and Airflow.

• Design cloud-based data platforms using AWS, Microsoft Azure, or Google Cloud Platform (GCP).

• Architect modern data solutions using platforms such as Databricks, Snowflake, BigQuery, Amazon Redshift, or Azure Synapse.

• Establish standards and best practices for data modeling, data governance, data quality, security, and metadata management.

• Work closely with data engineers, data scientists, analysts, application teams, and business stakeholders to translate requirements into scalable technical solutions.

• Evaluate existing data architecture and identify opportunities for modernization, optimization, automation, and cost reduction.

• Design solutions for batch and real-time/streaming data processing.

• Ensure high availability, fault tolerance, disaster recovery, and performance of data platforms.

• Provide technical leadership and mentoring to data engineers and development teams.

Preferred Candidate Profile :

• 8 - 10 years of experience in data engineering, big data, cloud data platforms, or data architecture, with significant experience designing enterprise-scale data solutions.

• Strong understanding of Big Data architecture and distributed computing concepts.

• Hands-on experience with Apache Spark and at least one major big data ecosystem such as Hadoop/Hive.

• Strong experience with Python, Java, or Scala.

• Experience designing and implementing ETL/ELT and data pipelines.

• Strong knowledge of Kafka or other event-streaming technologies for real-time data processing.

• Experience with one or more cloud platforms: AWS, Azure, or GCP.

• Experience with modern data platforms such as Databricks, Snowflake, BigQuery, Redshift, or Azure Synapse.

• Strong knowledge of SQL, data modeling, data warehousing, data lakes, and lakehouse architecture.

• Understanding of data governance, data quality, security, metadata, and master data concepts.

• Experience with orchestration tools such as Airflow, Azure Data Factory, AWS Glue, or similar technologies.