Infrastructure/ Platform Engineer Apache

Experis - ManpowerGroup • TRAFFIC • London, Not Specified, United Kingdom • 6h ago

Role Title: Infrastructure/ Platform Engineer - Apache
Duration: 9 Months
Location: Remote
Rate: £ - Umbrella only

Would you like to join a global leader in consulting, technology services and digital transformation?

Our client is at the forefront of innovation to address the entire breadth of opportunities in the evolving world of cloud, digital and platforms.

Role purpose / summary

? Refactor prototype Spark jobs into production-quality components, ensuring scalability, test coverage, and integration readiness.
? Package Spark workloads for deployment via Docker/Kubernetes and integrate with orchestration systems (e.g., Airflow, custom schedulers).
? Work with platform engineers to embed Spark jobs into InfoSum's platform APIs and data pipelines.
? Troubleshoot job failures, memory and resource issues, and execution anomalies across various runtime environments.
? Optimize Spark job performance and advise on best practices to reduce cloud compute and storage costs.
? Guide engineering teams on choosing the right execution strategies across AWS, GCP, and Azure.
? Provide subject matter expertise on using AWS Glue for ETL workloads and integration with S3 and other AWS-native services.
? Implement observability tooling for logs, metrics, and error handling to support monitoring and incident response.
? Align implementations with InfoSum's privacy, security, and compliance practices.
Required Skills and Experience:
? Proven experience with Apache Spark (Scala, Java, or PySpark), including performance optimization and advanced tuning techniques.
? Strong troubleshooting skills in production Spark environments, including diagnosing memory usage, shuffles, skew, and executor behavior.
? Experience deploying and managing Spark jobs in at least two major cloud environments (AWS, GCP, Azure).
? In-depth knowledge of AWS Glue, including job authoring, triggers, and cost-aware configuration.
? Familiarity with distributed data formats (Parquet, Avro), data lakes (Iceberg, Delta Lake), and cloud storage systems (S3, GCS, Azure Blob).
? Hands-on experience with Docker, Kubernetes, and CI/CD pipelines.
? Strong documentation and communication skills, with the ability to support and coach internal teams.
Key Indicators of Success:
? Spark jobs are performant, fault-tolerant, and integrated into InfoSum's platform with minimal overhead.
? Cost of running data processing workloads is optimized across cloud environments.
? Engineering teams are equipped with best practices for writing, deploying, and monitoring Spark workloads.
? Operational issues are rapidly identified and resolved, with root causes clearly documented.
? Work is delivered with a high level of independence, reliability, and professionalism

All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. We thank you for your interest and the time taken to apply!

Related Jobs

Gas Breakdown Engineer Ongoing Contract

Mirai Group • TRAFFIC • London, Not Specified, United Kingdom • 6h ago

Engineer

Join a Leading Gas Service Team with Mirai Group Exciting Opportunity for Self-Employed Gas Engineers! Mirai Group, a top-rated Energy Recruitment Agency, is seeking experiencedSelf-Employed Gas Service & Repair Engi...

6h ago

Apply

Senior Software Engineer, Exposure

P2P • TRAFFIC • London, Not Specified, United Kingdom • 12h ago

Engineer

Location London, United Kingdom Employment Type Full time Location Type Remote Department R&D Investigations The engineering team at Chainalysis is inspired by solving the hardest technical challenges and creating pr...

12h ago

Apply

Software Dev Engineer II, Cross Border Seller Programs

Amazon • TRAFFIC • London, Not Specified, United Kingdom • 12h ago

Engineer

Software Dev Engineer II, Cross Border Seller Programs Stay local, shop global! Amazon Cross Border (XB) team allows customers to easily discover a wide selection of international products available from Amazon's website...

12h ago

Apply

Lead Backend Engineer - Django

Kingfisher plc • TRAFFIC • London, Not Specified, United Kingdom • 12h ago

Engineer

Overview We're Kingfisher, a team of over 82,000 people bringing Kingfisher brands (B&Q, Screwfix, Brico Depot, Castorama and Koctas) to life. We aim to become the leading home improvement company and grow the larges...

12h ago

Apply

Senior Platform Engineer

Omnea Limited • TRAFFIC • London, Not Specified, United Kingdom • 1d ago

Engineer

Our Mission The way businesses buy stuff is completely broken . Whether it's SaaS, hardware, or contractors, the average B2B purchase takes over 3 months , requires 50+ emails , and involves multiple different stakeholde...

1d ago

Apply

Senior Security Engineer

Omnea Limited • TRAFFIC • London, Not Specified, United Kingdom • 1d ago

Engineer

1d ago

Apply

Principal Software Engineer

Outsource UK lTD • TRAFFIC • London, Not Specified, United Kingdom • 1d ago

Engineer

Job: Principal Software Engineer Company: BAE Systems Hourly Rate: £50.00 PAYE or £67.47 Umbrella Location : Barrow-in-Furness - 1 day per month on-site Contract : 12 months (Opportunity to be extended) Working Hours : 3...

1d ago

Apply

Software Engineer

Opus Recruitment Solutions Ltd • TRAFFIC • London, Not Specified, United Kingdom • 1d ago

Engineer

I'm looking for a Go, AWS, Node.js engineer to join a consultancy on a contracting basis. Inside IR35 contract. Experience: - GitHub Actions (Build & Deploy pipelines) - Networking - AWS - Terraform - Node JS - Docke...

1d ago

Apply

Devops Engineer - Outside IR35

Onyx-Conseil • TRAFFIC • London, Not Specified, United Kingdom • 1d ago

Engineer Outside IR35

Overview AWS DevOps Engineer (Contact Centre) - Outside IR35. This role supports a contact centre platform, maintaining and evolving the infrastructure, tooling, and automation that underpin the platform's development an...

1d ago

Apply

Graduate DevOps Engineer

TradingHub • TRAFFIC • London, Not Specified, United Kingdom • 1d ago

Engineer

About TradingHub Founded in 2010, we have grown from a united vision shared between two people to a team of over 140 across London, Toronto, New York and Singapore. We have achieved scale by building the best-in-class su...

1d ago

Apply