Support Engineer - AI Factory

Role : Support Engineer - AI Factory Network & ConnectivityAbout the Role :

You will be part of the Tier 2 support team responsible for the day-to-day health of CPT1 AI Factory, a 2,048-GPU NVIDIA H200 cluster built for large-scale AI training and inference. This role is biased toward network and connectivity engineering : keeping the east-west InfiniBand fabric and the north-south Ethernet/WAN paths in peak condition so that GPU training jobs run without interruption. You will work a 247 shift rotation alongside the NOC at Africa Data Centres and escalate to L2/L3 engineering when needed.

Key Responsibilities :

• Monitor, triage, and resolve network-layer incidents across the InfiniBand NDR 400G compute fabric (NVIDIA QM9700 switches) and the Ethernet storage/WAN fabric (NVIDIA Spectrum-4 SN5600, Arista 7280R3 WAN routers).

• Respond to Tier 1 alerts within the defined SLA windows : 4 hours for non-critical, 1 hour for degraded service, and 15 minutes for critical outages.

• Own first-response troubleshooting for connectivity faults : link flaps, NCCL all-reduce degradation, RDMA errors, IB subnet manager issues, and BGP/IP path failures on the LIT MPLS uplinks.

• Perform health checks on the 6 100G WAN links, validate path diversity, and monitor utilisation against the 600G aggregate capacity target.

• Support CSP direct-connect circuits, Azure ExpressRoute, AWS Direct Connect, GCP Partner Interconnect, Oracle Fast Connect provisioned via Teraco.

• Operate and maintain perimeter security : FortiGate 4400 firewall rules, stateful inspection policies, IDPS and TLS inspection where enabled.

• Administer the Rafay orchestration platform for bare-metal network provisioning : VRF/VLAN assignment, bond interfaces, cloud-init templates.

• Maintain and update NOC runbooks; create and close tickets in the incident management system with accurate RCA notes.

• Participate in planned maintenance windows (max 4 hrs/month) and ensure 7-day advance notifications are issued to customers.

• Collaborate with L2/L3 engineers and OEM (HPE/NVIDIA) support on complex IB fabric or WAN routing issues.

Required Skills & Experience :

• Having an engineering degree is a must or MCA or MCS with science in 12th STD.

• 3+ years in a networking or NOC role in a datacentre or HPC/cloud environment.

• Solid understanding of InfiniBand architecture : fat-tree topologies, NDR/HDR speeds, subnet managers (OpenSM/UFM), adaptive routing, RDMA/RoCE, and SHARP in-network compute.

• Hands-on experience with high-speed Ethernet switching : ECMP, VLAN, LAG/MLAG, BGP/OSPF, and MACSEC-encrypted WAN circuits.

• Familiarity with NVIDIA ConnectX NIC configuration (mlxconfig), IB diagnostics (ibstat, ibdiagnet, perfquery), and NCCL test suites.