Senior AI Test Engineer (Sydney or Melbourne)

Automic Group is Australia’s market leader in technology-driven investor administration. Combining proprietary cloud-native technology with expert service, Automic simplifies investor administration and governance for companies and funds. We provide the market's leading registry, employee share plans, IR analytics, and fund administration services for over 1,400 ASX-listed and unlisted organisations.

Automic’s culture is collaborative, inclusive, and purpose driven. It is grounded in our mission to connect companies and funds to their investors through technology, service, and trust, and guided by our values of One Automic, Go Beyond, Empowered, and Accountable.

About the Role

We're looking for a Senior AI Test Engineer to provide independent assurance that Automic’s AI-enabled products, agents and platforms are reliable, secure and production-ready.

Working closely with our AI Engineering team, you’ll shape quality strategy, establish robust AI evaluation and testing practices, and provide evidence-based release and residual-risk advice. You’ll play a key role in ensuring our AI solutions deliver safe, reliable and high-quality outcomes.

This role requires direct experience testing AI systems. Using AI coding assistants or generative AI to write test cases, scripts or documentation alone does not meet this requirement.Key Responsibilities include:

-
Lead risk-based quality strategies for AI applications, agents, retrieval systems and supporting services.

-
Deliver functional and non-functional assurance across integrations and end-to-end workflows.

-
Establish AI evaluation datasets, test data, quality thresholds and automated regression coverage.

-
Assess AI response quality and safety, including hallucination, groundedness, adversarial behaviour and sensitive-data exposure.

-
Validate agent workflows, authentication, authorisation, human approvals and tenant isolation.

-
Drive automated quality controls, defect management and continuous improvement across delivery pipelines.

-
Provide evidence-based release recommendations and investigate production quality issues.

About you!

To be successful in this role, you will have:

-
Hands-on experience designing and running tests or evaluations for generative AI, LLM, agentic or RAG applications, with clear examples of your personal contribution. This is essential; using AI tools to assist conventional software testing alone is not sufficient.

-
Proven experience in quality engineering or test automation for complex production systems.

-
Practical experience evaluating AI response accuracy, relevance, groundedness, hallucinations and safety.

-
Experience building evaluation datasets, defining quality thresholds and automating regression tests that account for variation in AI outputs.

-
Programming experience in Python or another language used for test automation and service testing.

-
Experience testing APIs, integrations and end-to-end workflows, including security, resilience and performance.

-
Strong analytical and communication skills, with the ability to explain findings, work through defects with engineers and make evidence-based release recommendations.

-
A tertiary qualification in technology, engineering or a related discipline, or equivalent professional experience.

Experience in financial services or another regulated environment is desirable.

Why join Automic:

You’ll be part of a team where operational excellence, collaboration, and continuous improvement are genuinely valued. This is an opportunity to deepen your expertise, contribute to meaningful outcomes, and grow your career within a high-performing, purpose-led fintech environment.

Key Benefits include:

-
Hybrid working

-
Professional career growth & development

-
Supportive & fast paced environment

-
Opportunity to grow with the business

If the above sounds like you, please apply below.