Software Developer (Content Integration & Data Pipelines)
- Compensation
- Not disclosed
- Location
- Canada
- Work setup
- Remote
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Software engineering
- Posted
Skills
Must have
- Education and experience: A degree in Computer Science or a related field, or equivalent practical experience, with at least three years of professional software development experience.
- TypeScript and JavaScript proficiency: Strong programming skills in TypeScript or JavaScript, particularly for content integration, parsing, and data processing.
- Backend development experience: Solid experience building backend services with Java or a comparable programming language.
- Production web scraping expertise: Hands-on experience developing and operating web scrapers and crawlers in production environments, with an understanding of robots.txt, rate limiting, responsible cr
- Event-driven architecture: Practical experience with event-driven systems and message-streaming technologies, particularly Kafka.
- Content processing skills: Knowledge of HTML parsing, content extraction, data transformation, and normalization techniques.
- Data pipeline engineering: Experience designing or maintaining high-volume, fault-tolerant data pipelines that process and deliver information reliably.
- Database and data modeling knowledge: Familiarity with SQL and data modeling principles.
- Development tools and environments: Experience working in Linux environments and using Git and modern software development workflows. Familiarity with Maven is also expected.
- Search technology knowledge: Experience with Solr, including indexing, schema design, and query tuning, is an asset.
- Relevant industry experience: Experience building content ingestion pipelines for news, media, or similar data-intensive applications is strongly preferred.
- System design capabilities: Understanding of distributed systems, scalability trade-offs, reliability, and performance optimization.
- Ownership and initiative: Ability to work independently, take responsibility for projects, and proactively identify opportunities for technical improvement.
- Analytical problem-solving: Strong troubleshooting skills and the ability to investigate complex technical issues and develop practical solutions.
- Communication and teamwork: Ability to communicate effectively with technical and non-technical stakeholders, collaborate across teams, and share knowledge.
- Interest in innovation: Enthusiasm for data engineering, automation, emerging technologies, and AI-powered capabilities.
Full description
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Developer (Content Integration & Data Pipelines) based in Canada.
We are seeking a skilled Software Developer to build and scale production-grade data pipelines that collect, process, enrich, and deliver online news and web content across a sophisticated media intelligence platform. In this role, you will help connect thousands of content sources to search, analytics, and AI-powered capabilities used by communications professionals. You will work across web scraping, content integration, backend development, and event-driven processing to ensure information is delivered accurately and efficiently. Using technologies such as TypeScript, Java, Kafka, and Solr, you will contribute to the architecture and evolution of high-volume content ingestion systems. You will play an active role in improving pipeline reliability, performance, observability, and data quality while following responsible web crawling practices. This remote opportunity offers the chance to solve complex engineering challenges, influence technical direction, and collaborate with a team focused on innovation, automation, and scalable data solutions.
Accountabilities
-
Build scalable web scrapers and crawlers: Develop and maintain production-ready scraping and crawling solutions that collect online content responsibly, following standards such as robots.txt, rate limiting, and applicable terms of use.
-
Develop content integration services: Implement parsing, extraction, and normalization logic using TypeScript and JavaScript to transform content from diverse sources into consistent, usable data.
-
Design event-driven pipelines: Build and maintain Kafka-based services that process, enrich, and route content in near real time, supporting reliable and scalable data flows.
-
Integrate external content sources: Connect third-party content providers, RSS and Atom feeds, and external APIs to expand and maintain content coverage.
-
Develop backend services: Design and implement Java-based RESTful services that support content processing, integration, and delivery across the platform.
-
Improve data quality and reliability: Establish and apply best practices for error handling, source management, validation, data normalization, and fault-tolerant pipeline operations.
-
Monitor and optimize pipeline performance: Implement observability and monitoring practices, investigate processing issues, and improve system performance, reliability, and operating costs.
-
Contribute to technical architecture: Participate in system design discussions, evaluate scalability trade-offs, and help define the technical direction of content acquisition and data processing systems.