Lead Databricks Engineer - London - £500 - £600

Databricks Lead Engineer - Hybrid/London - £500 - £600 per day - 6 months - Banking

Role Overview

We are hiring a hands-on Databricks Lead Engineer that has experience delivering modern data platforms on Databricks. This role requires a minimum of 2 years of Databricks experience gained recently (ie, current/recent projects using modern platform capabilities).

NB Initially this contract is 4/5 days per week onsite, before moving to Hybrid working pattern after 2/3 months

Databricks Lead Engineer - Hybrid/London - £500 - £600 per day - 6 months - Banking

Key Responsibilities

• Lead the engineering delivery of Databricks lakehouse solutions from ingestion to curated serving layers.
• Define and implement Medallion Architecture (Bronze/Silver/Gold) and reusable engineering patterns.
• Build scalable ingestion pipelines using AutoLoader, Lakeflow Connect, batch/streaming, and incremental patterns.
• Develop Declarative Pipelines with Expectations (DLT) to enforce and monitor data quality.
• Implement and operate Unity Catalog for governance, access control, lineage, and secure data sharing patterns.
• Drive code quality and operational excellence (CI/CD approach, testing strategy, monitoring, incident triage).
• Partner with architects, platform teams, and stakeholders to align delivery with enterprise standards.
• Mentor engineers and act as the technical escalation point during delivery.

Minimum Experience (Filter Criteria)

• 2+ years of hands-on Databricks experience, in recent years (eg, within the last 2-3 years), demonstrating usage of modern Databricks capabilities and patterns.
• Evidence of production delivery (not training-only or lab-only exposure).

Must Have (Non-Negotiable)

• Databricks Certification (active/current): at least one of:
• Databricks Certified Data Engineer Associate/Professional
• Databricks Certified Machine Learning Associate/Professional
• Databricks Certified Generative AI Engineer (Associate)
• Unity Catalog hands-on experience:
• Metastore/catalog design, grants, lineage, and secure access patterns.
• Declarative Pipelines with Expectations (DLT):
• Building pipelines, defining expectations, handling failures/quarantines, observability.
• Ingestion engineering using Databricks-native approaches:
• AutoLoader and/or Lakeflow Connect, streaming and incremental ingestion patterns.
• Medallion Architecture implementation and best practices:
• Designing and implementing Bronze/Silver/Gold with practical decisions (schema evolution, CDC/upserts, SCD patterns, performance strategy).
• (If the candidate is operating in an architect capacity, they must be able to lead the design-not just follow an existing blueprint.)

Should Have

• Demonstrable use of latest Databricks capabilities (candidate can explain what they used recently and why).
• Strong Databricks engineering fundamentals:
• Delta Lake (MERGE, schema enforcement/evolution, OPTIMIZE/ZORDER, VACUUM)
• Databricks Workflows/job orchestration
• Production-grade PySpark/SQL
• Clear understanding of pipeline reliability:
• Observability, alerting, replay/backfill strategies, and operational runbooks.

Nice to Have

• Lakeflow ingestion connectors (specific connector experience is a plus).
• RBAC/masking implementations (row-level security, column masking, sensitive data handling) using Unity Catalog.
• GenAI on Databricks:
• Mosaic AI, Vector Search, model serving, RAG pipelines, AI Functions.
• Lakebase awareness or hands-on experience.
• Workload/query optimisation:
• Photon usage, cluster sizing, shuffle/skew mitigation, caching strategy, partitioning, file sizing.
• Cost awareness and controls:
• Understanding DBU drivers, job vs all-purpose compute, cluster policies, monitoring and chargeback/showback patterns.

Databricks Lead Engineer - Hybrid/London - £500 - £600 per day - 6 months - Banking

This temporary contract is inside IR35 and will require working under the direction of the client delivery manager as part of a multi-disciplinary team. The successful candidate will follow established delivery processes and working practices

Damia Group Limited acts as an employment agency for permanent recruitment and employment business for the supply of temporary workers. By applying for this job you accept our Data Protection Policy which can be found on our website.

Please note that no terminology in this advert is intended to discriminate on the grounds of a person's gender, marital status, race, religion, colour, age, disability or sexual orientation. Every candidate will be assessed only in accordance with their merits, qualifications and ability to perform the duties of the job.

Should the role require the successful candidate to undergo and be eligible for UK Security Vetting. Clearance sponsorship will be provided where required. Due to the nature of the work, candidates should meet the relevant residency requirements. If applicable, Reserved Post nationality restrictions will be confirmed by the client. Damia is committed to inclusive recruitment and welcomes applicants from all backgrounds.

Damia Group is acting as an Employment Business in relation to this vacancy and in accordance to Conduct Regulations 2003.