Software Engineer (Big data/Python/AWS/Databricks)

Software Engineer

Location: Urbandale, IA, 50322

Duration: 14 Months on W2 contract

Job Description

We are looking for a highly technical engineer to build and support weather data acquisition, processing, and analysis products, including our weather service, using a cloud-based solutions stack. We are open to early or advanced career candidates with strong examples of novel contributions and highly independent work in a fast-paced software delivery environment.

Essential attributes/experience:

• Excellent coding skills (Python) with production software deployment experience
• Big data experience (terabyte or petabyte level data sources)

Core understanding of cloud computing (e.g., AWS services like IAM, Lambda, S3, RDS)

Example Responsibilities (including but not limited to):

• Architect and propose new AWS/Databricks solutions & updates to existing backend systems that ingest, cache, and process terabyte and petabyte level weather data (e.g., from IBM Weather Company (API)
• Work closely with the product management team and end users to understand customer experience and system requirements, build backlog, and prioritizework
• Build infrastructure as code (e.g., Terraform)Improve system scalability and optimize workflows to reduce cloud costs
• Create and update REST APIs and backend processes (e.g., AWS Chalice/Lambda) for current, historical, and forecast weather
• dataBuild/support solutions involving containerization (e.g., Docker), caching (Redis), and databases (e.g., PostgreSQL/PostGIS)
• Support Databricks/Spark pipelines for historical weather data backfill into S3 (Parquet/Delta Lake) and data analysis (SQL/PySpark)
• Migrate and maintain CI/CD pipelines (e.g., GitHub Actions)
• Enhance monitoring and alerting for weather systems (e.g., Datadog)
• Enable field testing and customer support operations by debugging and fixing data issues

Work with data scientists to scalably fetch and manipulate large weather data sets for models and analysis