Lead Data Engineering (Hands-on Python / Databricks)

Job Title: Technical Delivery Lead - Data Engineering Lead (Hands-on Python / Databricks)
Contract: 6 months - Potentially extension.
Location: London Hybrid (5 days in 10 in the office)
Rate: Highly competitive
Status: Inside IR35
Working pattern: Full time

We are seeking an experienced Technical Delivery Lead with strong expertise in Python, Databricks, and Apache Spark to lead the successful delivery of enterprise-scale data engineering initiatives

This is a hands-on leadership role combining technical delivery, solution design, and data engineering expertise. The successful candidate will lead the delivery of a greenfield data platform while remaining actively involved in development activities.

Essential Skills

• Proven experience in Data Engineering or Software Engineering.
• Previous experience in a Technical Lead, Lead Data Engineer, or Technical Delivery Lead position.
• Strong hands-on expertise with Python.
• Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake.
• Strong experience with Apache Spark /PySpark.
• Proven track record building and optimising large-scale ETL/ELT pipelines.
• Strong understanding of modern data architecture and Lakehouse principles.
• Excellent SQL skills and experience working with distributed data systems.
• Experience working with cloud platforms such as AWS, Azure, or GCP.
• Strong Agile delivery experience.
• Experience leading technical teams and delivering complex projects.

Expected split: 30-50% hands-on coding + technical problem-solving

Key Responsibilities

• Lead end-to-end delivery of data engineering solutions.
• Design and build scalable data pipelines using Databricks and Azure.
• Develop and optimise solutions using Python and PySpark.
• Drive engineering best practices, code reviews, and technical standards.
• Work closely with architecture, business, and technology stakeholders.
• Mentor and support engineers within the team.
• Contribute to data platform and Lakehouse architecture decisions.

What We're Looking For

• Approximately 60% hands-on engineering and 40% technical leadership.
• Proven experience leading data engineering teams.
• Strong stakeholder management and communication skills.
• Experience delivering greenfield data platform initiatives.

This is an excellent opportunity to join a high-profile programme with significant scope for extension and future team growth.

Why Join Us?

• Work on innovative projects that make a difference in the financial sector.
• Collaborate with a talented team in a vibrant environment.
• Enjoy a competitive daily rate and flexible working arrangements.

If you are ready to take your career to the next level and contribute to exciting projects, apply now! We can't wait to meet you!

Pontoon is an employment consultancy. We put expertise, energy, and enthusiasm into improving everyone's chance of being part of the workplace. We respect and appreciate people of all ethnicities, generations, religious beliefs, sexual orientations, gender identities, and more. We do this by showcasing their talents, skills, and unique experience in an inclusive environment that helps them thrive. If you require reasonable adjustments at any stage, please let us know and we will be happy to support you.