What are the responsibilities and job description for the Senior Data Engineer position at Diligent Tec, Inc?
Position: Senior Data Engineer (PySpark Databricks Oracle/PostgreSQL Migration)
Location: Alpharetta, GA
Job type: Contract
ONLY W2
Interview Levels: 2 Rounds
1st Round-Webex Video
2nd round-Coding F2F
Exp Level: Min 10 Years
Expert-level proficiency in PySpark and Databricks.
Primary Skill Focus: PySpark, Databricks (Hands-on coding test expected)
Skills Required:
1. RDBMS Expertise:
Strong hands-on experience with Oracle and PostgreSQL databases.
Deep understanding of database architecture, design, indexing, and performance tuning.
Expert-level SQL skills: queries, stored procedures, functions, triggers, and views.
2. Database Migration
Proven experience in end-to-end database migration projects (preferred: Oracle → PostgreSQL).
Strong ability to perform data mapping, transformation, validation, and reconciliation.
Experience using migration tools, scripts, and automation frameworks.
3. Data Engineering & Analysis
Advanced proficiency in Databricks for large-scale data processing.
Expert in PySpark and Python for data transformation and analytics.
Ability to build, enhance, and optimize complex ETL/ELT data pipelines.
4. Job Scheduling & Automation
Experience creating and maintaining Databricks jobs for scheduled reporting.
Familiarity with workflow orchestration tools (Airflow, ADF, Step Functions, etc.).
5. Performance Optimization
Strong background in performance tuning for Oracle and PostgreSQL.
Experience with index strategies, query optimization, execution plan analysis, and caching.
6. Cloud Platforms
Good understanding of AWS Cloud, including RDS, S3, EMR, Lambda, Glue, or similar services.
Location: Alpharetta, GA
Job type: Contract
ONLY W2
Interview Levels: 2 Rounds
1st Round-Webex Video
2nd round-Coding F2F
Exp Level: Min 10 Years
Expert-level proficiency in PySpark and Databricks.
Primary Skill Focus: PySpark, Databricks (Hands-on coding test expected)
Skills Required:
1. RDBMS Expertise:
Strong hands-on experience with Oracle and PostgreSQL databases.
Deep understanding of database architecture, design, indexing, and performance tuning.
Expert-level SQL skills: queries, stored procedures, functions, triggers, and views.
2. Database Migration
Proven experience in end-to-end database migration projects (preferred: Oracle → PostgreSQL).
Strong ability to perform data mapping, transformation, validation, and reconciliation.
Experience using migration tools, scripts, and automation frameworks.
3. Data Engineering & Analysis
Advanced proficiency in Databricks for large-scale data processing.
Expert in PySpark and Python for data transformation and analytics.
Ability to build, enhance, and optimize complex ETL/ELT data pipelines.
4. Job Scheduling & Automation
Experience creating and maintaining Databricks jobs for scheduled reporting.
Familiarity with workflow orchestration tools (Airflow, ADF, Step Functions, etc.).
5. Performance Optimization
Strong background in performance tuning for Oracle and PostgreSQL.
Experience with index strategies, query optimization, execution plan analysis, and caching.
6. Cloud Platforms
Good understanding of AWS Cloud, including RDS, S3, EMR, Lambda, Glue, or similar services.