Scala Data Engineer
Location: Hybrid in New Jersey
Type: Full-Time Role
Experience: Mid - Senior Level
Salary Range: 80,000 - 120,000
About ITI
ITI is an industry-leading information and data management solutions firm with business-driven technology designed for the world's largest banks, financial services and healthcare companies. Headquartered in One World Trade Center at the heart of New York City's Financial District, ITI has over 300 employees across 10 branch and satellite offices in key global, financial services markets.
Description
We are looking for a Scala Solutions Developer and Data Engineer capable of building a data standardized solution on the Hadoop cluster for one of our Fortune 500 client programs. This is a high visibility, fast-paced key initiative that will integrate data across internal and external sources, provide analytical insights and integrate with the customer's critical systems.
Job Duties

• As a Scala Developer, the candidate will be responsible for developing and implementing scalable data processing solutions using Apache Spark.
• The ideal candidate should have experience in both batch and real-time data processing, along with a strong background in technologies such as Databricks, Snowflake, Kafka, SQL and Unix.
• The candidate will work closely with our data engineering and data science teams to design and optimize data pipelines and ensure efficient and reliable data processing.
• Develop and execute data pipeline testing processes and validate business rules and policies.
• Ability to design & build real-time applications using Apache Kafka & Spark Streaming.
• Ability to build abstracted, modularized reusable code components.
• Collaborating with cross-functional teams to integrate Scala solutions with other systems and technologies.
• Collaborating with data engineering and data science teams to understand data requirements and design optimal data pipelines.
• Writing efficient Scala jobs and optimizing Spark code for performance and scalability.
• Developing and optimizing SQL queries for data extraction, transformation, and loading.
• Produce unit tests for Scala transformations and helper methods.
• Ensuring data quality and integrity throughout the data processing pipelines.
• Monitoring and troubleshooting data processing issues and implementing necessary fixes.
• Keeping up-to-date with the latest advancements in Scala and big data technologies

Requirements

• Expertise in handling complex large-scale Big Data environments preferably (5Tb ).
• Minimum 5 years of experience in building data pipelines using both Scala structured streaming and Scala batch processes.
• At least 4 years of experience in Designing, developing, and maintaining data processing solutions using Apache Spark (Scala).
• Experience working in both on-prim Hadoop cluster as well as Databricks environment.
• Strong understanding of Functional programming and RESTful APIs
• Experience with design and implementation of ETL framework for complex warehouses
• Knowledge of large data sets and experience with performance tuning and troubleshooting

We would love to be able to contact everyone personally; however, this is not possible. If you have not heard from us within 6 weeks of submitting your application, we thank you for your interest in joining ITI Data, but on this occasion, your application was not shortlisted.
Please send your resumes to
#jobs #itidata