Big Data Developer

Role : Big Data Engineer Spark / Scala

Location : Jacksonville FL (3 days onsite)

Contract role

Looking for 9 Years

What is in it for you?

We are looking for a highly skilled Spark/Scala Senior Engineer with 8 to 10 years of experience in designing, developing, and optimizing large-scale batch and real-time data processing solutions. The ideal candidate will possess strong expertise in Scala, Apache Spark, Snowflake, Kafka/NiFi, and modern Data Engineering frameworks. This role requires hands-on technical leadership, performance optimization expertise, and the ability to mentor junior team members while driving engineering best practices

Job Description

 Design, develop, and optimize large-scale batch and streaming data pipelines using Scala and Apache Spark.

· Build robust data processing solutions using Spark SQL, DataFrames, and Datasets.

· Develop and maintain Spark Structured Streaming applications for real-time data ingestion and processing.

· Integrate data pipelines with enterprise data platforms such as Snowflake.

· Implement scalable data ingestion frameworks utilizing Apache Kafka and/or Apache NiFi.

Perform Spark performance tuning, including:

· Partition optimization

· Caching strategies

· Shuffle optimization

· Broadcast joins

· Resource allocation tuning

· Develop clean, maintainable, and testable Scala code following functional programming principles.

· Participate in code reviews and establish coding standards and best practices.

· Create automated tests and support CI/CD pipeline improvements.

· Monitor production data pipelines and troubleshoot performance or data quality issues.

· Conduct root cause analysis and implement preventive measures.

· Lead technical and architectural discussions across teams.

· Mentor junior engineers and provide technical guidance.

· Collaborate with cross-functional stakeholders to deliver scalable data solutions.

Required Technical Skills

Scala

• Strong hands-on experience with Scala development.
• Deep understanding of:
• Case Classes
• Traits
• Pattern Matching
• Collections Framework
• Implicits
• Strong knowledge of Functional Programming concepts:
• Immutability
• Higher-Order Functions (HOFs)
• Monads
• Experience developing and maintaining production-grade Scala applications.

Apache Spark

• Strong expertise with:
• Spark SQL
• DataFrames
• Datasets
• Extensive experience with Spark Structured Streaming.
• Deep understanding of Spark internals:
• DAG Execution
• Lazy Evaluation
• Shuffle Operations
• Stage Execution
• Experience optimizing large-scale Spark workloads.
• Hands-on experience with:
• Parquet
• Avro
• ORC
• Delta Lake
• Apache Kafka
• Apache NiFi

Snowflake

• Experience designing and querying Snowflake data warehouse schemas.
• Familiarity with:
• Snowflake Stages
• File Formats
• Streams
• Tasks
• Knowledge of Snowpark for Scala is preferred.

Data Engineering & Development

• Strong SQL and query optimization skills.
• Experience with Git-based version control workflows.
• Familiarity with Docker and containerized application development.
• Experience working within Agile development environments.