Data Engineer (Cloud)

About the role

The Bank’s data‑driven strategy requires a Cloud Data Engineer to design, build and optimise petabyte‑scale data pipelines. You will transform high‑volume, heterogeneous raw data into governed, low‑latency data products that power advanced analytics, machine learning and real‑time insights.

Key responsibilities

• Architect, develop and maintain scalable, fault‑tolerant pipelines using Ab Initio, Python/Scala, Apache Spark, Microsoft Fabric and Databricks.
• Implement batch and streaming workflows, ensuring high availability, cost efficiency and reliability for the enterprise Data Warehouse and Data Lake.
• Apply Data‑Ops practices – automation, testing and orchestration – for end‑to‑end data flows.
• Collaborate with software engineers to create secure APIs for real‑time data consumption.
• Manage a hybrid‑cloud data platform on Azure (Fabric, Synapse, Databricks, SAS) and on‑prem technologies (DB2, Netezza, Denodo).
• Define infrastructure‑as‑code, observability and service‑level objectives for all data components.
• Work with Information Security and Data Governance to enforce POPIA‑compliant data masking and privacy controls.
• Lead technical evaluations, proofs of concept and integration of emerging data engineering tools.

Required profile

• Strong background in designing and operating large‑scale data pipelines in cloud environments.
• Experience with both batch and low‑latency streaming data processing.
• Proven ability to collaborate across cross‑functional teams and align technical solutions with business goals.
• Knowledge of data security, governance and privacy regulations such as POPIA.

Required skills

• Ab Initio
• Python
• Scala
• Apache Spark
• Microsoft Fabric
• Databricks
• Azure (Fabric, Synapse)
• SAS
• DB2
• Netezza
• Denodo
• Infrastructure as Code