The role

G-Research is seeking a DevOps focused Infrastructure Engineer, with deep experience in HDFS, YARN and the Hadoop ecosystem, to join its growing Big Data Platform Engineering team.

Big Data Platform Engineering manages a significant distributed Cloudera HDFS storage footprint, has recently adopted Airflow and has also started to explore other Open Source offerings, such as Ozone. In addition, there is also an overlap in technology support within the team, in products such as Trino, Spark, Grafana, Hive, Nifi and HBase to name a few.

As a Senior Big Data Infrastructure Engineer, you will help us deliver automation, feature enhancements and best practice security on our existing hybrid cloud platforms, as well as explore new technologies and integration with other Open Source products within our ecosystem.

There is a significant focus on configuration management, container orchestration, Infrastructure as Code (IaC) and CI/CD. In return, you will gain exposure to the latest hardware and software technologies in a forward-thinking company, which values innovation, personal development and training.

Key responsibilities of the role include:

• Developing, deploying and maintaining big data solutions within the Hadoop ecosystem, such as HDFS, YARN, Ranger and Hive.
• Delivering engineering solutions ensuring they are scalable, stable, performant, and operationally efficient
• Providing metrics, documentation, and self-service infrastructure to help our customers work at pace and get the most out of the platforms
• Proactively identifying and addressing system performance bottlenecks, security vulnerabilities, and areas for improvement to ensure systems are stable and robust
• Deploying and maintaining Airflow
• Participating in the out of hours rota, providing support and maintenance for our critical systems across the BigData ecosystem

Who are we looking for?

Whatever your technical background, good coding ability is essential for this role.

The ideal candidate will have the following skills and experience:

• Strong HDFS, YARN and Hadoop ecosystem knowledge, preferably on the Cloudera Distribution but also OSS Distributions
• Excellent coding skills, preferably Python
• Experience working with CI\CD pipelines, such as Jenkins and ArgoCD
• Experience working with Configuration Management tools, such as Ansible and Terraform
• Strong Linux, Hardware and Networking knowledge
• Experience deploying and running applications on Docker and Kubernetes, including the creation of Helm charts
• Experience working with distributed systems

Highly desirable but not essential:

• Experience working with Ozone
• Experience working with other Big Data technologies, such as Spark, Trino, Hive, Airflow, Ranger, Oozie etc
• Data Engineering experience (Spark application development, SQL)
• Team Lead experience

Why should you apply?

• Market-leading compensation plus annual discretionary bonus
• Informal dress code and excellent work/life balance
• Excellent paid time off allowance of 25 days
• Sick days, military leave, and family and medical leave
• Generous 401(k) plan
• 16-weeks' fully paid parental leave
• Medical and Prescription, Dental, and Vision insurance
• Life and Accidental Death & Dismemberment (AD&D) insurance
• Employee Assistance and Wellness programs
• Generous relocation allowance and support
• Great selection of office snacks, and hot and cold drinks
• On-site gym and car parking