What are the responsibilities and job description for the DataBricks Engineer - Insider Risk position at RSC Solutions?
Role Description
- We are currently seeking a talented and motivated Senior Data Engineer to join our Insider Risk team.
- This role will be responsible for designing, developing & deploying cloud data solutions for risk, regulatory, finance & compliance projects and be at the forefront of on-prem to cloud migration initiative.
- This role will require indirect leadership and mentorship of junior team members.
- Work with business teams, product managers, and other stakeholders to understand business requirements, needs, or process gaps that would be addressed via proposed applications.
- Engage with IT systems architects, database administrators, and cyber security team to meet the back-end functionality demands of the application.
- Liaise with other technology and development teams as needed to coordinate on the integration of applications with the larger company software ecosystem.
- Participate in periodic development meetings to align development priorities and objectives, assign tasks, and share experiences and challenges with applications under development
Role Objectives: Interpersonal
Work with business units as a main point of contact to diagnose and support database design and query issues, as well as provide training on database use and functions. Partner with IT system administrators to perform maintenance and upgrades on database infrastructure, update security lists, and troubleshoot performance and hardware issues. Collaborate with application development team to advise on capacity and resource requirements, maintain data exchange and API structures, monitor job schedules and operate reporting tools. Engage with network administrators to maintain telecommunication structures, network security policy and access lists, and troubleshoot any connectivity issues.
Role Objectives: Expertise
- Extensive experience on Cloud solutions, specifically in Azure
- Experience with Azure cloud services, Azure Data Factory, Gen 2, Azure Databases, Functions, Databricks, etc
- Strong knowledge of Python, Java or similar high-level server-side languages
- Advanced working SQL knowledge on relational databases, preferably on MS SQL Server and databases on Azure cloud
- Experience with the DevOps lifecycle (git, Jenkins, etc.)
- Experience with Jira/Confluence
- Experience with REST web services and microservice architecture
- Databricks Cluster configuration, Unity catalog repository configuration.
- Build large-scale batch and real-time data pipelines with data processing frameworks in Azure cloud platform.
- Partner with architects, engineers, information analysts, business, and technology stakeholders for developing and deploying enterprise grade platforms that enable data-driven solutions.
- Coordinating with platform, APIM, Azure DevOps, and Support.
- Analyze and design technical solutions on Azure to address business needs.
- Develop, test, and modify software to improve efficiency of data platforms and applications.
- Integrating data from a variety of sources, assuring that they adhere to data quality and accessibility standards.
- Enable streaming data use-cases and standard design patterns in a big data environment.
- Review and provide feedback on core code changes and support production deployment.
Qualifications and Skills
Minimum 4 years of software development experience in enterprise applications, preferably in the financial services industry and on data-heavy systems
Bachelor's or master's degree in computer science or information systems preferred.