Pyspark Data Engineer

IO Associates • England, England, GB • 1d ago

PySpark Data Engineer | up to £450/day Inside | Remote with occasional London travel

We are seeking a PySpark Data Engineer to support the development of a modern, scalable data lake for a new strategic programme. This is a greenfield initiative to replace fragmented legacy reporting solutions, offering the opportunity to shape a long-term, high-impact platform from the ground up.

Key Responsibilities:
* Design, build, and maintain scalable data pipelines using PySpark 3/4 and Python 3.
* Contribute to the creation of a unified data lake following medallion architecture principles.
* Leverage Databricks and Delta Lake (Parquet format) for efficient, reliable data processing.
* Apply BDD testing practices using Python Behave and ensure code quality with Python Coverage.
* Collaborate with cross-functional teams and participate in Agile delivery workflows.
* Manage configurations and workflows using YAML, Git, and Azure DevOps.

Required Skills & Experience:
* Proven expertise in PySpark 3/4 and Python 3 for large-scale data engineering.
* Hands-on experience with Databricks, Delta Lake, and medallion architecture.
* Familiarity with Python Behave for Behaviour Driven Development.
* Strong understanding of YAML, code quality tools (e.g. Python Coverage), and CI/CD pipelines.
* Knowledge of Azure DevOps and Git best practices.
* Active SC clearance is essential - applicants without this cannot be considered.

Contract Details:
* 6-month initial contract with long-term extension potential (multi-year programme).
* Inside IR35.

This is an excellent opportunity to join a high-profile programme at its inception and help build a critical data platform from the ground up. If you are a mission-driven engineer with a passion for scalable data solutions and secure environments, we'd love to hear from you.

Related Jobs

M365 Security Design Engineer

ARM • TRAFFIC • Bedfordshire, Luton, United Kingdom • 1d ago

Engineer Outside IR35

M365 Security Design Engineer Remote working - UK Wide 6-month contract Paying up to 90p/h (Outside IR35) - DOE Responsibilities : Design and implement security controls across E5 Microsoft 365 services (e.g., Endpoints;...

1d ago

Apply

Data Cabling Engineer

Onsite Recruitment Ltd • TRAFFIC • Staffordshire, Stoke-on-trent, United Kingdom • £180 / day • 1d ago

Engineer

The On-Site Group are looking for Data Cabling Engineer in Stoke on Trent. Duties will include general cabling installing phone lines and CCTV. You will need you own car/van for this role - due to sites being in Sheffiel...

1d ago

Apply

Security Engineer

V7 Recruitment • TRAFFIC • Derby, City, United Kingdom • £33 - £35 / day • 1d ago

Engineer Outside IR35

Security Service Engineer (Temporary long term contract) V7 are looking for Warden call maintenance engineers for an a exciting long term contract opportunity. With this contract as a service engineer you will have the c...

1d ago

Apply

Digital Manufacturing Engineer

Alexander Mann Solutions - Contingency • TRAFFIC • Derby, City, United Kingdom • 1d ago

Engineer

We are AMS. We are a global total workforce solutions firm; we enable organisations to thrive in an age of constant change by building, re-shaping, and optimising workforces. Our Contingent Workforce Solution (CWS) is on...

1d ago

Apply

Software Engineer

Line Up Aviation • TRAFFIC • Hertfordshire, Stevenage, United Kingdom • £60 - £75 / day • 1d ago

Engineer

On behalf of our client, we are seeking to recruit someone as a Software Engineer for 12 months. As the Software Engineer you will join the delivery team undertaking the development of an application-level C++ Engine whi...

1d ago

Apply

Senior MDM Intune Infrastructure Engineer (SC Clearance)

NonStop Consulting • TRAFFIC • Manchester, City, United Kingdom • 1d ago

Engineer

Gain high value experience and impact the lives of millions! Join one of the largest projects in UK! If this sounds appealing to you then you are in luck as we are currently recruiting a Senior MDM Intune Infrastructure...

1d ago

Apply

115425 Platform Engineer

CBSbutler Holdings Limited trading as CBSbutler • TRAFFIC • Hampshire, Farnborough, United Kingdom • £600 - £625 / day • 1d ago

Engineer

Platform Engineer +On site in Farnborough +SC cleared role +Inside IR35 + 575 - 615 a day Skills: +Infrastructure as code +Openshift +Kubernetes +Ansible We are seeking an experienced Senior Platform Engineer to enable t...

1d ago

Apply

Electrical Maintenance Engineer

IO Associates • Cambridgeshire, England, GB • 1d ago

Engineer

Electrical Maintenance Engineer Corsham (Onsite) Contract Duration: Up to 9 months | Active SC Clearance required iO associates has partnered with highly regulated facility to find an electrical maintenance engineer who...

1d ago

Apply

Software Engineer

Certain Advantage • APPLICATION • Hertfordshire, Stevenage, United Kingdom • £75 / day • 1d ago

Engineer

World Class Defence Organisation is currently looking to recruit a Software Engineer Subcontractor on an initial 12 month contract. The role can be based from either the companies Stevenage or Bristol site, depending on...

1d ago

Apply

SC DevOps Engineer, £400 - £500 A Day Outside IR35, 6 Months

IO Associates • London, England, GB • 1d ago

Engineer Outside IR35

SC Cleared DevOps Engineer £400 - £500 Outside IR35 London/ Manchester (1 Day A Month On-site) 6 Month Contract A consultancy client of iO's have just won a project in the public sector space in which they will need seve...

1d ago

Apply