What are the responsibilities and job description for the DBT experienced AWS Data Engineer with Databricks position at HAN IT Staffing Inc.?
Role: DBT - experienced AWS Data Engineer with Databricks Location: Lebanon, NJ Contract position
Responsibilities
Design and Development of Data Pipelines:
Design, build, and optimize robust ETL/ELT pipelines using AWS services (S3, Glue, Lambda) and the Databricks platform (Spark, Delta Lake, DLT).
Ingest and process large volumes of structured and semi-structured data from various sources (APIs, databases, streaming platforms like Kafka or Kinesis) into a centralized data lake or lakehouse.
Data Transformation and Modeling:
Develop and maintain data models (e.g., star/snowflake schemas, medallion architecture) optimized for analytics and BI tools using dbt (Data Build Tool).
Write complex and efficient SQL queries and Python/PySpark code for data manipulation, transformation, and validation within the Databricks environment.
Implement data quality checks, tests, and documentation as part of the dbt workflow, enforcing data governance and security standards.
Orchestration and Automation:
Orchestrate and monitor data workflows using Databricks Jobs or external tools like AWS MWAA (Managed Workflows for Apache Airflow).
Implement CI/CD pipelines and version control (Git) for all data engineering artifacts (code, configurations, dbt models) to ensure reliable and consistent deployments.
Performance Optimization and Operations:
Monitor, troubleshoot, and resolve issues in production data pipelines and environments to ensure high performance, reliability, and cost-efficiency.
Tune Spark jobs and optimize Delta Lake features (Z-Order, partitioning) to handle growing data volumes and complexity.
Collaboration and Support:
Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
Provide expertise and guidance on data best practices, promoting a culture of data quality and governance.
Must Have skills
SQL DBT core and DBT Cloud AWS (redshift)
Data bricks with AWS SQL server DB
Stone branch scheduling tool
Should understand CICD, GIT
Work in Agile environment with JIRA
Other Skills required/ Good to have:
Tableau experience
Harness devops Proficient in Linux / Unix environments