Senior Databricks Architect/Lead Databricks Engineer

Remote (UK)

£700/day Outside IR35

Initial 12-Month Contract | Long-Term Programme | Poosible Extension

We're seeking an exceptional Senior Databricks Architect with a strong hands-on Data Engineering background to join a major long-term transformation programme.

This is not a pure architecture role. You'll be a Databricks specialist who remains close to the technology, designing and building enterprise-scale data solutions while driving engineering excellence across critical data platforms.

You'll take ownership of delivering robust, scalable, and production-grade data solutions within a modern Databricks ecosystem, ensuring platforms are governed, secure, maintainable, and optimised for long-term growth.

Key Responsibilities

Architect, build and optimise enterprise-grade data solutions within Databricks

Design and implement scalable Delta Lake data architectures following Medallion principles

Lead data engineering delivery across multiple strategic workstreams

Develop highly resilient, production-ready pipelines with strong operational controls

Drive data governance, security, lineage and compliance best practices

Collaborate with product, engineering and business teams to deliver trusted data assets

Mentor internal engineering teams and establish engineering standards

Core Workstreams

Product Data Ingestion

Design and maintain large-scale ingestion pipelines

Integrate assessment, content and product data into shared enterprise data platforms

Ensure data quality, consistency and operational reliability

AI & GenAI Feature Enablement

Build and maintain data foundations powering LLM and GenAI capabilities

Design schemas and pipelines supporting AI-driven products and document processing

Handle complex parsed, normalised and sensitive datasets at scale

Support emerging AI use cases including evaluation processing, bulk uploads and intelligent workflows

IEP & Medicaid Data Integration

Build robust integrations across IEP, Medicaid and associated service platforms

Deliver reconciliation, data quality and cross-system consistency

Support enterprise reporting and operational analytics requirements

Essential Experience

Deep expertise in Databricks and Delta Lake architecture

Proven experience implementing and operating Medallion Architecture (Bronze, Silver, Gold)

Advanced pipeline engineering experience including:

Orchestration

Idempotent processing

Error handling

Monitoring

Recovery strategies

Data reconciliation

Strong Unity Catalog experience including:

Data governance

Lineage

Fine-grained permissions

Row-level and column-level security

Extensive ETL and data integration expertise across complex enterprise systems

Strong understanding of:

Data quality frameworks

Data normalisation

Backfilling strategies

Data integrity controls

Experience handling sensitive and regulated data environments

Knowledge of PII protection techniques including:

Masking

Tokenisation

Anonymisation

Familiarity with GDPR and broader data privacy standards

Performance and cost optimisation expertise including:

Z-Ordering

Liquid Clustering

Data Skipping

Query tuning

Shuffle optimisation

Cloud cost control

8+ years software engineering experience

3+ years operating at Senior/Lead Data Engineer or Databricks Architect level

Highly Desirable

Databricks AI capabilities

Vector Search

Model Serving

LLM Pipelines

Retrieval-Augmented Generation (RAG)

EdTech domain experience

Experience with industry standards such as:

Ed-Fi

OneRoster

CEDS

1EdTech

High-impact programme combining Data Engineering, AI and Enterprise Architecture

Opportunity to shape the future data platform of a growing organisation

Significant ownership and influence from day one