Expert Data Engineer – Kafka, Python, API, SQL, ETL/ELT, Java

About the role

We are seeking an expert Data Engineer to design, build and maintain high‑performance, real‑time data platforms that support enterprise‑scale operations. The role focuses on creating resilient streaming solutions using Apache Kafka, Python and API‑driven ingestion, while ensuring data quality, security and scalability.

Key responsibilities

• Architect and implement event‑driven pipelines on Apache Kafka for real‑time data movement.
• Develop and maintain Python‑based ingestion services, webhooks and API integrations.
• Design and optimise SQL queries and data models for operational (OLTP) workloads.
• Build ETL/ELT processes, manage schema registries (Avro, Protobuf) and enforce data‑quality standards.
• Implement idempotent pipelines with retry, replay and compensation mechanisms.
• Collaborate with downstream teams to ensure compatibility with Java‑based stream processors and containerised deployments.

Required profile

• Proven experience delivering production‑grade streaming platforms in complex enterprise environments.
• Strong understanding of data modelling, metadata management and data lineage.
• Ability to normalise and correlate data from multiple sources such as CMDB, identity and security tools.
• Experience with performance tuning of relational databases and query optimisation.

Required skills

• Apache Kafka and event streaming
• Python programming
• API integration and webhooks
• SQL and relational database optimisation
• ETL/ELT pipeline design
• Schema registries (Avro, Protobuf)
• Java (for stream processing or connector development)
• Kubernetes and container orchestration
• Terraform infrastructure‑as‑code
• MongoDB or other NoSQL document stores
• SAP integration interfaces