Data Engineer

About the role

The Data Engineer designs, implements and maintains the data infrastructure used by the Insights team to analyse data and derive insights. Working with Engineering Leads, Data Architects and other stakeholders, the engineer ensures data is properly sourced, stored and accessible, and extracts maximum predictive value from data assets.

Key responsibilities

• Design, develop, test and maintain ETL data pipelines to extract, transform and load data into a data warehouse or data lake, including quality checks, optimisation and troubleshooting.
• Build and maintain lakehouse (medallion) pipelines, monitor ETL jobs and develop data connectors and integrations with databases, APIs and flat files.
• Design data models and database schemas, modernise legacy on-premises environments to cloud-native platforms such as Databricks, tune queries and implement data streaming and real-time processing.
• Build dashboards and BI reports (e.g. with PowerBI, Tableau or Pyramid Analytics), analyse data for trends and share insights with stakeholders.
• Ensure compliance with data privacy policies, maintain data governance policies and standards, audit data access, and implement disaster recovery and business continuity plans.
• Collaborate with Engineering Leads, Architects, Analysts and Scrum Masters, and communicate pipeline status and progress.

Required profile

• At least 3 years of relevant IT experience or related qualification.
• 3+ years of data pipeline design and development experience.
• Experience with production BI environments and tools such as PowerBI, Tableau or Pyramid Analytics.
• Advantageous: Databricks Certified Data Engineer Associate, Azure certifications, retail/finance/customer/HR domain experience, mentoring experience, and exposure to .NET Entity Framework.

Required skills

• Proficiency in Python and/or SQL; RDBMS experience (SQL Server, MySQL, PostgreSQL).
• Scripting with PowerShell, Bash or Python for automation; unit and integration testing.
• Advantageous: Azure data technologies, Azure DevOps, Git, CI/CD, Docker, Kafka, Hadoop, Delta Lake and PySpark.