DataBricks Data Engineer

Job Responsibilities
• Work closely with business stakeholders, analytics architects, and AI engineers to help exploratory and production application data preparation needs. Data customers will include human analysts, analytics engineers, AI and analytical models, dashboarding applications, and data stores.
• Set up pipelines to ingest, store, clean, merge, transform, and aggregate data from multiple external and internal data sources
• Set up both batch and streaming pipelines
• Build data pipelines and transformations in Azure Databricks

Required Skills
• Azure Databricks data engineering (Databricks Data Engineer Professional)
• Understanding of Databricks Unity Catalog (including Delta Lake/Delta tables)
• Understanding of structured, unstructured, and semi-structured data processing
• Understanding of financial services cyber security and fraud data domain
• Highly proficient Python and SQL skills
• PySpark and Spark SQL
Preferred Experience:
• 3 years of Databricks data engineering experience (Databricks Data Engineer Professional)
• 7 years of experience in data engineering pipelines for analytics—must include batch and streaming data
• 2 years working in a financial services or security related data domain

Key Talents
• Adaptability: Data engineering requirements will change frequently as analytical needs and priorities flex. The data engineer who fills this role must be able to understand and quickly adapt to new requests
• Conscientiousness: it is critical the person who fills this role takes in upon themselves to ensure that the output of their pipelines are what is practically needed by the business users and analytics engineers downstream. This person must be able to interpret specification in the context of business needs and work to clarify specifications to ensure that they serve the practical business need."