What are the responsibilities and job description for the Sr. Python/Spark Platform Engineer position at Motion Recruitment?
Job Description Software Engineer 3
Senior Cloud Engineer – Apache Spark / GCP / Python / Microservices (S3)
Overview
This role supports the Model Risk Management platform used to run statistical risk models and large-scale data workloads. The engineer will help maintain and enhance a cloud-native platform used by statisticians and data scientists working with millions of customer accounts. The platform runs on Google Cloud Platform and supports distributed data processing using Apache Spark.
Experience with large-scale datasets, GCP services, Spark, and containerized microservices is essential. Experience in financial services is preferred but not required.
Key Responsibilities
Platform Engineering
Deploy, configure, and maintain OpenShift clusters or GCP projects for Spark workloads
Support platform capabilities for statistical model execution
Distributed Data Processing
Design and implement large-scale data processing workflows using Apache Spark
Tune Spark jobs using Kubernetes orchestration and auto-scaling
Application & Tooling
Build Python/Django services and microservices supporting platform users
Configure tooling and internal applications for data processing and model execution
Automation & CI/CD
Build and maintain CI/CD pipelines using GitHub Actions, Sonar, Harness, Helm
Monitoring & Troubleshooting
Monitor Spark jobs and cluster health using Prometheus, Grafana, and GCP tools
Debug distributed systems and optimize resource utilization
Security & Compliance
Implement RBAC and encryption for data in transit and at rest
Collaboration
Work closely with cross-functional teams to define requirements and support deployments
Qualifications
Experience
3 years with Apache Spark
2 years Django development
1 years creating/maintaining conda environments
2 years with OpenShift/Kubernetes
Experience Working With GCP Or Another Major Cloud Provider
Technical Skills
Spark frameworks (PySpark, Scala, or Java)
OpenShift/Kubernetes administration
Docker container experience
Python, Scala, or Java programming
Experience with GitHub Actions, Helm, Harness
Knowledge of distributed systems and cloud storage (S3, GCS, HDFS)
Education
Bachelor’s degree in Computer Science, Engineering, or related field
Posted By: Steve Chatt
Senior Cloud Engineer – Apache Spark / GCP / Python / Microservices (S3)
Overview
This role supports the Model Risk Management platform used to run statistical risk models and large-scale data workloads. The engineer will help maintain and enhance a cloud-native platform used by statisticians and data scientists working with millions of customer accounts. The platform runs on Google Cloud Platform and supports distributed data processing using Apache Spark.
Experience with large-scale datasets, GCP services, Spark, and containerized microservices is essential. Experience in financial services is preferred but not required.
Key Responsibilities
Platform Engineering
Deploy, configure, and maintain OpenShift clusters or GCP projects for Spark workloads
Support platform capabilities for statistical model execution
Distributed Data Processing
Design and implement large-scale data processing workflows using Apache Spark
Tune Spark jobs using Kubernetes orchestration and auto-scaling
Application & Tooling
Build Python/Django services and microservices supporting platform users
Configure tooling and internal applications for data processing and model execution
Automation & CI/CD
Build and maintain CI/CD pipelines using GitHub Actions, Sonar, Harness, Helm
Monitoring & Troubleshooting
Monitor Spark jobs and cluster health using Prometheus, Grafana, and GCP tools
Debug distributed systems and optimize resource utilization
Security & Compliance
Implement RBAC and encryption for data in transit and at rest
Collaboration
Work closely with cross-functional teams to define requirements and support deployments
Qualifications
Experience
3 years with Apache Spark
2 years Django development
1 years creating/maintaining conda environments
2 years with OpenShift/Kubernetes
Experience Working With GCP Or Another Major Cloud Provider
Technical Skills
Spark frameworks (PySpark, Scala, or Java)
OpenShift/Kubernetes administration
Docker container experience
Python, Scala, or Java programming
Experience with GitHub Actions, Helm, Harness
Knowledge of distributed systems and cloud storage (S3, GCS, HDFS)
Education
Bachelor’s degree in Computer Science, Engineering, or related field
Posted By: Steve Chatt