2128 Sr Data Engineer - Platform Modernization & Migration
Fully remote data engineer executing a client data platform modernization: migrating ~1,000 datasets to Snowflake, upgrading Airflow 2 to Airflow 3, converting legacy Hive data to Apache Iceberg, and moving Kafka streaming across cloud regions. Core stack: Python, advanced SQL, Snowflake, Airflow, Spark/PySpark, Kafka, Iceberg, and AWS S3.
- Compensation
- Not disclosed
- Location
- Argentina
- Work setup
- Remote
- Seniority
- Senior
- Employment
- Contract
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Minimum 2+ years hands-on data engineering experience with large-scale datasets, Apache Airflow, and Apache Spark
- Strong software engineering fundamentals with hands-on Python development
- Advanced SQL skills (complex joins, CTEs, window functions, query tuning)
- Proven experience with Snowflake data modeling and schema architecture
- Direct experience building and testing pipelines with Apache Airflow, Apache Kafka, and Apache Spark/PySpark
- Expertise with Apache Iceberg and Apache Parquet data formats
- Experience with AWS S3 (data lakes, bucket partitioning, lifecycle policies)
- Ability to define testing strategies and independently debug complex distributed data pipelines
- Strong communication skills with a track record of working independently
- Must pass AWS WorkSpace connectivity test and submit screenshot of results with application
Nice to have
- Experience with enterprise Snowflake migrations (warehouse modernization, cost and performance tuning)
- Direct experience with Apache Airflow platform migrations (version upgrades, executor changes, platform moves)
- Hands-on experience using AI-assisted coding/development tools
Full description
Position: Data Engineer (Platform Modernization & Migration) Location: Fully Remote (EMEA & APAC Preferred) Contract Type: Full-time, Permanent Time Zone Alignment: Requires overlap until at least 12:00 PM Pacific Time (PT)
🧭 About Inallmedia.com
Inallmedia.com is a global technology and design firm focused on building impactful digital solutions through remote, distributed teams. We partner with international clients across industries, providing long-term technical expertise, product innovation, and team augmentation.
🚀 Project Overview
You will join a core Data Engineering organization, working in close partnership with the Data Platform team to drive a major data platform modernization initiative. The primary focus of this project is executing pre-designed migration blueprints for approximately 1,000 datasets to Snowflake with Airflow 3 orchestration. You will play a crucial role in accelerating time-sensitive infrastructure goals by evolving legacy Lakehouse formats to Apache Iceberg, modernizing streaming infrastructure across cloud regions, and adapting existing pipelines to uphold strict data quality and performance standards.
🔍 Key Responsibilities
- Pipeline Migration & Upgrades: Execute the end-to-end migration of approximately 1,000 datasets to Snowflake, including upgrading orchestration workflows from Apache Airflow 2 to Airflow 3.
- Storage & Data Lake Modernization: Migrate legacy Hive data structures and datasets to Apache Iceberg format using the Snowpark Connect framework/connectors.
- Streaming Infrastructure & Region Migration: Reconfigure, update, and migrate Confluent/Apache Kafka streaming connections across cloud regions (e.g., migrating from US East 1 to US East 2).
- Pipeline Adaptation & Performance Tuning: Adapt existing data pipelines to target Snowflake environments while optimizing SQL queries (utilizing CTEs, window functions, and complex joins) to ensure high performance and data reliability.
- Workflow Scheduling & Monitoring: Manage, test, and debug DAGs (Directed Acyclic Graphs) and workflow scheduling within Apache Airflow.
- Testing & Quality Assurance: Implement testing strategies and conduct hands-on code debugging to guarantee data quality and consistency throughout the platform modernization.
💡 Must-Have Skills
- Industry Experience: Minimum 2+ years of hands-on data engineering experience working with large-scale datasets, Apache Airflow, and Apache Spark.
- Core Languages & SQL: Strong software engineering fundamentals with hands-on Python development experience and advanced SQL skills (complex joins, CTEs, window functions, and query tuning).