What are the responsibilities and job description for the Lead Software Engineer – Databricks / PySpark / AWS position at JPMorgan Chase?
Job Summary:
We have an exciting and rewarding opportunity for you to take your software engineering career to the next level.
As a Lead Software Engineer at JPMorgan Chase within the Corporate Technology, you play a crucial role in an agile team dedicated to enhancing, building, and delivering trusted, market-leading technology products in a secure, stable, and scalable manner. As a key technical contributor, you are tasked with implementing critical technology solutions across multiple technical domains, supporting various business functions to achieve the firm's business objectives.
Job Responsibilities:
- Design solutions at the appropriate level of detail and drive consensus among peers as needed.
- Champion software engineering best practices within the team.
- Collaborate with software engineers and cross-functional teams to design and implement deployment strategies using AWS Cloud and Databricks pipelines.
- Lead the design, development, testing, and implementation of application solutions.
- Partner with technical experts, stakeholders, and team members to resolve complex technical challenges.
- Proactively address issues to support leadership objectives and prevent customer impact.
- Design, develop, and maintain robust data pipelines for ingesting, processing, and storing large volumes of data from diverse sources.
- Implement ETL (Extract, Transform, Load) processes to ensure data quality and integrity using tools such as Apache Spark and PySpark.
- Monitor and optimize the performance of data systems and pipelines.
- Apply best practices for data storage, retrieval, and processing.
- Maintain comprehensive documentation for data systems, processes, and workflows.
Required Qualifications, Capabilities, and Skills:
- Formal training or certification on software engineering concepts and 5 years applied experience.
- Proficiency in programming languages such as Python and PySpark.
- Experience designing and implementing data pipelines in cloud environments.
- Strong background in design, architecture, and development using AWS Services, Databricks, Spark, Snowflake, and related technologies.
- Experience with CI/CD tools such as Jenkins, GitLab, or Terraform.
- Familiarity with containerization and orchestration technologies including ECS, Kubernetes, and Docker.
- Ability to troubleshoot issues related to Big Data and Cloud technologies.
Preferred Qualifications, Capabilities, and Skills:
- 5 years of experience leading and developing data solutions in AWS Cloud.
- 10 years of experience building, implementing, and managing data pipelines using Databricks on Spark or similar cloud technologies