Job Title: Lead Databricks Engineer

Location: NYC, NY, Iselin, NJ || Hybrid

Job Type: Contract

Job Description:

Major Responsibilities:

· Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo

· Design, develop, and deploy Databricks jobs to process and analyze large volumes of data.

· Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.

· Optimize Databricks jobs for performance and scalability to handle big data workloads.

· Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.

· Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.

· Demonstrated proficiency with Data Analytics, Data Insights

· Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process

· Azure Synapse/Bigquery/Redshift is good to have.

· Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.

Skills:

· 5 years’ – Strong experience in Finance / Banking industry – Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo

· 10 years - Enterprise Data Management

· 10 years - SQL Server based development of large datasets

· 5 years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding.

· Azure Synapse/Bigquery/Redshift experience is good to have

· 3 years Python(numpy, pandas) coding experience

· Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have

· Experience in Data warehousing - OLTP, OLAP, Dimensions, Facts, and Data modeling.

· Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skills

· Capable of discussing enterprise level services independent of technology stack

· Experience with Cloud based data architectures, messaging, and analytics

· Superior communication skills

· Cloud certification(s)

· Any experience with Regulatory Reporting is a Plus.

Job Type: Contract

Salary: $36.68 - $98.45 per hour

Benefits:

• 401(k)
• Dental insurance
• Health insurance

Experience level:

• 10 years

Schedule:

• 8 hour shift
• Day shift
• Monday to Friday

Experience:

• Data Bricks: 10 years (Required)
• Banking: 6 years (Required)

Work Location: On the road