What are the responsibilities and job description for the Spark Data Engineer position at Bespoke Technologies, Inc.?
BT-454 – Spark Data Engineer
Location: Tysons, VA
Required Skills
Location: Tysons, VA
Required Skills
- Demonstrates experience building and managing data pipelines
- Demonstrates experience with Python
- Demonstrates experience with cloud computing, using AWS services
- Demonstrates experience processing data using Apache Spark
- Demonstrates experience with an RDBMS (PostgreSQL, Oracle, MySQL) and writing SQL queries
- Demonstrates experience with Linux and shell scripting
- Demonstrates experience analyzing data in different file formats like CSV, XML, JSON, Avro, Parquet, etc.
- Demonstrates experience writing and validating unit tests
- Experience with NiFi, Apache AirFlow, or an equivalent solution or tool for orchestrating data pipelines
- Experience with Java or Scala
- Experience administering an EMR/Spark cluster
- Experience conducting performance tuning of a Spark job
- Experience developing cloud-based security solutions
- Experience following a configuration management process to review and deploy code as part of releases