Software Engineer

• Excellent coding skills (Python) with production software deployment experience
• Big data experience (terabyte or petabyte level data sources)
• Core understanding of cloud computing (e.g., AWS services like IAM, Lambda, S3, RDS)

• Architect and propose new AWS/Databricks solutions & updates to existing backend systems that ingest, cache, and process terabyte and petabyte level weather data (e.g., from IBM Weather Company API)
• Work closely with the product management team and end users to understand customer experience and system requirements, build backlog, and prioritize work
• Build infrastructure as code (e.g., Terraform)
• Improve system scalability and optimize workflows to reduce cloud costs
• Create and update REST APIs and backend processes (e.g., AWS Chalice/Lambda) for current, historical, and forecast weather data
• Build/support solutions involving containerization (e.g., Docker), caching (Redis), and databases (e.g., PostgreSQL/PostGIS)
• Support Databricks/Spark pipelines for historical weather data backfill into S3 (Parquet/Delta Lake) and data analysis (SQL/PySpark)
• Migrate and maintain CI/CD pipelines (e.g., GitHub Actions)
• Enhance monitoring and alerting for weather systems (e.g., Datadog)
• Enable field testing and customer support operations by debugging and fixing data issues
• Work with data scientists to scalably fetch and manipulate large weather data sets for models and analysis

Salary : $60 - $65