Databricks Data Engineer

Harnham - Data & Analytics Recruitment • TRAFFIC • London, Not Specified, United Kingdom • £500 - £600 / * • 13h ago

Databricks Data Engineer

£500 - £600 per day

Outside IR35

We're partnering with a leading online retail company that's transforming the way data and real-time intelligence shape customer experiences. Their mission is to harness cutting-edge data and streaming technologies to drive smarter decisions, improve efficiency, and create personalised journeys for millions of shoppers worldwide.

The Role

As a Senior Data Engineer, you'll play a key role in developing and optimising the backbone of the company's data platform. You'll be responsible for building and maintaining large-scale, real-time data pipelines that power analytics, machine learning, and operational systems across the business.

You'll collaborate with software engineers, data scientists, and analytics teams to ensure the platform delivers reliable, high-quality, and compliant data at scale. This is a hands-on engineering role that blends software craftsmanship with data architecture expertise.

Key responsibilities:

Design and implement high-throughput data streaming solutions using Kafka, Flink, or Confluent.
Build and maintain scalable backend systems in Python or Scala, following clean code and testing principles.
Develop tools and frameworks for data governance, privacy, and quality monitoring, ensuring full compliance with data protection standards.
Create resilient data workflows and automation within Airflow, Databricks, and other modern big data ecosystems.
Implement and manage data observability and cataloguing tools (e.g., Monte Carlo, Atlan, DataHub) to enhance visibility and reliability.
Partner with ML engineers, analysts, and analytics engineers to understand their data needs and enable advanced data use cases.
Contribute to an engineering culture that values testing, peer reviews, and automation-first principles.

What You'll Bring

Strong experience in streaming technologies such as Kafka, Flink, or Confluent.
Advanced proficiency in Python or Scala, with a solid grasp of software engineering fundamentals.
Proven ability to design, deploy, and scale production-grade data platforms and backend systems.
Familiarity with data governance frameworks, privacy compliance, and automated data quality checks.
Hands-on experience with big data tools (Airflow, Databricks) and data observability platforms.
Collaborative mindset and experience working with cross-functional teams including ML and analytics specialists.
Curiosity and enthusiasm for continuous learning - you stay up to date with the latest tools and trends in data engineering and love sharing knowledge with others.

Please send your email

Related Jobs

Senior Automation Engineer

83zero Ltd • TRAFFIC • London, Not Specified, United Kingdom • £300 - £400 / * • 11h ago

Engineer Outside IR35

Senior Automation Engineer - Asset Management London Hybrid 12-18 Months Contract - Outside IR35 We're partnering with a leading organisation in the Asset Management space that's seeking a Senior Automation Engineer for...

11h ago

Apply

Platform Data Engineer

Harnham - Data & Analytics Recruitment • TRAFFIC • London, Not Specified, United Kingdom • £500 - £600 / * • 13h ago

Engineer Outside IR35

Senior Platform Engineer £500-£600 per day Outside IR35 6-months Join one of the UK's leading online retailers as they evolve their next-generation data platform. This is an opportunity to shape the backbone of a modern...

13h ago

Apply

Part Time M365 IT & Security Support Engineer £300 - £350 Outside IR35 Onsite

Opus Recruitment Solutions Ltd • TRAFFIC • London, Not Specified, United Kingdom • £300 - £350 / * • 1d ago

Engineer Outside IR35

Part Time M365 IT & Security Support Engineer £300 - £350 Outside IR35 Onsite We're recruiting an experienced M365 IT & Security Support Engineer to assist our client through a busy transitional period. This 2-mo...

1d ago

Apply

Part Time | M365 IT & Security Support Engineer | £300 - £350 | Outside IR35 | Onsite

Opus Recruitment Solutions Ltd • ATS • London, Not Specified, United Kingdom • £300 - £350 / * • 1d ago

Engineer Outside IR35

Part Time | M365 IT & Security Support Engineer | £300 - £350 | Outside IR35 | Onsite We're recruiting an experienced M365 IT & Security Support Engineer to assist our client through a busy transitional period. T...

1d ago

Apply

Contract Data Engineer - eDV Cleared

Searchability NS&D • TRAFFIC • London, Not Specified, United Kingdom • £450 - £650 / * • 1d ago

Engineer Outside IR35

CONTRACT DATA ENGINEER - eDV CLEARED NEW OUTSIDE IR35 CONTRACT OPPORTUNITY AVAILABLE WITHIN A LEADING NATIONAL SECURITY SME FOR A DATA ENGINEER WITH eDV CLEARANCE Contract job opportunity for a Data Engineer National Sec...

1d ago

Apply

Contract Senior Software Engineer - eDV Cleared

Searchability NS&D • TRAFFIC • London, Not Specified, United Kingdom • £500 - £750 / * • 1d ago

Engineer Outside IR35

CONTRACT SENIOR SOFTWARE ENGINEER - eDV CLEARED NEW CONTRACT JOB OPPORTUNITY AVAILABLE WITHIN A LEADING NATIONAL SECURITY COMPANY FOR A SENIOR SOFTWARE ENGINEER WITH eDV CLEARANCE Contract job opportunity for a Senior So...

1d ago

Apply

Contract Palantir Foundry Engineer - DV Cleared

Searchability NS&D • TRAFFIC • London, Not Specified, United Kingdom • £500 - £650 / * • 1d ago

Engineer Outside IR35

CONTRACT PALANTIR DATA ENGINEER - DV CLEARED NEW OUTSIDE IR35 CONTRACT OPPORTUNITY AVAILABLE WITHIN A HIGHLY SECURE DELIVERY PROGRAMME FOR A DV CLEARED PALANTIR FOUNDRY DATA ENGINEER. Contract opportunity for a Palantir...

1d ago

Apply

Security and IT Systems Administrator / IT Support Engineer - SC Cleared Environment

MLC Partners • TRAFFIC • London, Not Specified, United Kingdom • £200 / * • 2d ago

Engineer Outside IR35

IT Systems & Security Administrator to support and manage a secure MS digital infrastructure environment. This role requires active SC clearance and experience in Microsoft 365 and Azure environments. Outside IR35. L...

2d ago

Apply

Databrick Platform Engineer - Outside IR35 Contract

Tenth Revolution Group • TRAFFIC • London, Not Specified, United Kingdom • £400 - £500 / * • 3d ago

Engineer Outside IR35

Databricks Platform Engineer - Outside IR35 Contract Please note - this role will require you to be based in the UK with the unrestricted right to work in the UK. This organisation is not able to offer sponsorship. An ex...

3d ago

Apply

AI Engineer

Robert Half • TRAFFIC • London, Not Specified, United Kingdom • 3d ago

Engineer Outside IR35

Robert Half Technology are assisting a market-leading pharmaceuticals organisation to recruit an AI Engineer on a contract basis. Outside IR35. Hybrid working - London based. Role The AI Engineer will design, develop, an...

3d ago

Apply