What are the responsibilities and job description for the AWS Data engineer with Databricks & DBT - Onsite NJ position at E-Solutions, Inc.?
Job Title : AWS Data Engineer with Databricks and DBT
Job Location : Iselin, NJ
Skill Cluster : Data-AWS Glue with EMR, Spark, Redshift,Kinesis,S3,AWS Native Data Services.
Primary Skill : CICS
Job Description
Responsibilities
- Design and Development of Data Pipelines : Design, build, and optimize robust ETL/ELT pipelines using AWS services (S3, Glue, Lambda) and the Databricks platform (Spark, Delta Lake, DLT).
- Ingest and process large volumes of structured and semi-structured data from various sources (APIs, databases, streaming platforms like Kafka or Kinesis) into a centralized data lake or lakehouse.
Data Transformation and Modeling:
- Develop and maintain data models (e.g., star/snowflake schemas, medallion architecture) optimized for analytics and BI tools using dbt (Data Build Tool).
- Write complex and efficient SQL queries and Python/PySpark code for data manipulation, transformation, and validation within the Databricks environment.
- Implement data quality checks, tests, and documentation as part of the dbt workflow, enforcing data governance and security standards.
- Orchestration and Automation: Orchestrate and monitor data workflows using Databricks Jobs or external tools like AWS MWAA (Managed Workflows for Apache Airflow).
- Implement CI/CD pipelines and version control (Git) for all data engineering artifacts (code, configurations, dbt models) to ensure reliable and consistent deployments.
- Performance Optimization and Operations: Monitor, troubleshoot, and resolve issues in production data pipelines and environments to ensure high performance, reliability, and cost-efficiency.
- Tune Spark jobs and optimize Delta Lake features (Z-Order, partitioning) to handle growing data volumes and complexity.
Collaboration and Support:
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
- Provide expertise and guidance on data best practices, promoting a culture of data quality and governance.
- Must Have skills SQLDBT core and DBT Cloud AWS (redshift)Data bricks with AWS SQL server DBStone branch scheduling tool
- Should understand CICD, GITWork in Agile environment with JIRAOther
Skills required/ Good to have: Tableau experience Harness devops Proficient in Linux / Unix environments
|
Shubham Saxena w: e: “Disclaimer: E-Solutions Inc. provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws. We especially invite women, minorities, veterans, and individuals with disabilities to apply. EEO/AA/M/F/Vet/Disability.” |
AWS Data engineer with Databricks & DBT1AWS,Databricks,DBTN/AC2CUnited States