What are the responsibilities and job description for the Sr. Databricks Engineer - Hybrid NYC position at Expert Technology Services?
Job Summary (Senior Data Engineer)
- Design, build, and optimize scalable batch and real-time data pipelines for enterprise analytics, machine learning, and AI-driven solutions.
- Develop and maintain data ingestion, transformation, and processing frameworks for various data types (structured, semi-structured, unstructured).
- Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion design patterns (Bronze, Silver, Gold).
- Ensure data quality, governance, lineage, security, and compliance across complex data ecosystems.
- Develop distributed data processing and streaming solutions with PySpark, Spark SQL, Kafka, and AWS Kinesis.
- Architect and deploy cloud-based data solutions on AWS, leveraging services like S3, EMR, EC2, Athena, Redshift, Lambda, and more.
- Support migration from traditional Hadoop and EMR environments to modern cloud-native platforms.
- Build and automate orchestration, scheduling, and CI/CD pipelines using Airflow, Databricks Workflows, and related frameworks.
- Collaborate with architects, data scientists, analysts, and business stakeholders to deliver enterprise-grade solutions.
- Design and implement Generative AI-powered and AI-assisted solutions to enhance engineering productivity and platform intelligence.
- Evaluate and integrate emerging AI technologies including LLMs, RAG, Vector Databases, and Model Context Protocol (MCP).
- Maintain high standards for system reliability, scalability, performance, and cost optimization.
- Contribute to the modernization and innovation of data platforms, driving adoption of cloud-native and AI-driven architectures.
Ideal Candidate Will Have:
- 8 years in data/software/big data engineering roles.
- Deep expertise with Python, PySpark, Spark SQL, Databricks, Delta Lake, Snowflake, AWS, and Kafka.
- Strong experience with distributed computing, data modeling, and large-scale data processing.
- Hands-on experience with CI/CD, orchestration, and automation frameworks.
- Knowledge of data governance, metadata management, and data quality frameworks.
- Experience with Generative AI, LLMs, RAG, and developer productivity tooling.
- AWS certifications and exposure to supply chain, retail, healthcare, or manufacturing data domains are a plus.
- Design, build, and optimize scalable batch and real-time data pipelines for enterprise analytics, machine learning, and AI-driven solutions.
- Develop and maintain data ingestion, transformation, and processing frameworks for various data types (structured, semi-structured, unstructured).
- Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion design patterns (Bronze, Silver, Gold).
- Ensure data quality, governance, lineage, security, and compliance across complex data ecosystems.
- Develop distributed data processing and streaming solutions with PySpark, Spark SQL, Kafka, and AWS Kinesis.
- Architect and deploy cloud-based data solutions on AWS, leveraging services like S3, EMR, EC2, Athena, Redshift, Lambda, and more.
- Support migration from traditional Hadoop and EMR environments to modern cloud-native platforms.
- Build and automate orchestration, scheduling, and CI/CD pipelines using Airflow, Databricks Workflows, and related frameworks.
- Collaborate with architects, data scientists, analysts, and business stakeholders to deliver enterprise-grade solutions.
- Design and implement Generative AI-powered and AI-assisted solutions to enhance engineering productivity and platform intelligence.
- Evaluate and integrate emerging AI technologies including LLMs, RAG, Vector Databases, and Model Context Protocol (MCP).
- Maintain high standards for system reliability, scalability, performance, and cost optimization.
- Contribute to the modernization and innovation of data platforms, driving adoption of cloud-native and AI-driven architectures.
Ideal Candidate Will Have:
- 8 years in data/software/big data engineering roles.
- Deep expertise with Python, PySpark, Spark SQL, Databricks, Delta Lake, Snowflake, AWS, and Kafka.
- Strong experience with distributed computing, data modeling, and large-scale data processing.
- Hands-on experience with CI/CD, orchestration, and automation frameworks.
- Knowledge of data governance, metadata management, and data quality frameworks.
- Experience with Generative AI, LLMs, RAG, and developer productivity tooling.
- AWS certifications and exposure to supply chain, retail, healthcare, or manufacturing data domains are a plus.