Vidaa
Senior Data Engineers
Senior Data Engineer in Singapore who owns end-to-end real-time data ingestion and real-time data warehouse systems — building streaming ETL pipelines, tuning Kafka/Spark/Flink jobs, and delivering real-time metrics and dashboards for business teams. Cloud data tooling like BigQuery, DBT, and Databricks is core to the stack.
- Compensation
- Not disclosed
- Location
- Singapore, Singapore
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 3+ years
- Category
- Not specified
- Posted
Skills
Must have
- Bachelor's degree or above in Computer Science, Software Engineering, Data Science, or related majors
- 3+ years of professional big data development experience in the internet industry
- Proven experience in end-to-end real-time data warehouse implementation and data ingestion system construction
- Proficient in English reading, writing and verbal communication, able to collaborate with overseas teams
- Expert-level knowledge of Kafka, including partition strategy, consumer mechanism, and diagnosing message backlog, data skew, loss and duplication
- Hands-on experience with Spark, Spark SQL, Flink and Flink SQL, including streaming task development, resource tuning, latency and throughput optimization
- Familiar with real-time data warehouse layered architecture and dimensional modeling methodologies
- Deep understanding of big data ingestion systems: real-time collection, CDC synchronization, data cleaning, and exception handling for multi-source data
- Practical experience with cloud-native data tools including BigQuery, DBT and Databricks
- Proficient in at least one of Python, Scala and Java
- Familiar with Linux commands and Shell scripting for task deployment, log analysis and daily operations
- Familiar with mainstream relational databases and Binlog/CDC real-time synchronization principles
- Knowledgeable in real-time data SLA governance, data quality monitoring, alert configuration and incident review
Nice to have
- Overseas study or work experience
- Experience with Google Cloud, Azure or other mainstream cloud big data platforms
- Overseas team collaboration and real-time data governance experience
Full description
Job Responsibilities
- Own the construction and maintenance of the company’s business data ingestion system. Responsible for the full lifecycle development of real-time collection, cleaning, and warehousing of business logs, user behavioral data, and database operational data, ensuring the completeness, timeliness, and stability of end-to-end data ingestion pipelines.
- Lead the implementation and iterative optimization of the real-time data warehouse architecture. Design multi-layer real-time data models, develop and tune streaming ETL pipelines, and support the stable output of real-time metrics, dashboards, and business monitoring data.
- Collaborate closely with business teams, data analysts, and data product managers. Fully understand business logic and statistical calibers, independently complete data requirement decomposition, logical sorting, development implementation, release verification, and continuous iteration.
- Maintain and optimize Kafka streaming pipelines for real-time data ingestion and consumption. Troubleshoot and resolve core online issues including message backlog, data skew, data loss and duplication, and ensure high availability and stability of real-time data pipelines.
- Develop, tune, and operate Spark real-time tasks. Continuously optimize computing resource utilization, data latency, and throughput, and improve the overall performance and stability of streaming data pipelines.
- Participate in data standardization construction, real-time data quality governance, and pipeline redundancy optimization. Summarize and standardize data ingestion specifications and development best practices to improve the overall team’s data development efficiency.
- Provide stable real-time data support for downstream business scenarios including BI dashboards, data services, user portrait platforms, and business effectiveness measurement, ensuring reliable online data service delivery.
Job Requirements
- Bachelor’s degree or above in Computer Science, Software Engineering, Data Science, or related majors. 3+ years of professional big data development experience in the internet industry, with proven experience in end-to-end real-time data warehouse implementation and data ingestion system construction.
- Proficient in English reading, writing and verbal communication skills, capable of collaborating with overseas teams and business stakeholders. Overseas study or work experience is highly preferred.
- Expert-level knowledge of Kafka. Familiar with real-time data ingestion, partition strategy, consumer mechanism, and capable of diagnosing and optimizing online risks such as message backlog, data skew, data loss and duplication; experienced in large-scale streaming pipeline stability governance.