What are the responsibilities and job description for the Senior Data Engineer - Databricks position at PEN Software Inc.?
Job Title: Senior Data Engineer (Databricks, Spark, Kafka, AWS)
Work Location: Hybrid – Atlanta, GA (Local candidates preferred)
Employment Type: Contract (6 months with potential extension)
W2 / C2C / 1099 via PEN Software
Experience Range: 6 – 12 years total experience
Minimum 5 years in Data Engineering
Job Summary
PEN Software is seeking a Senior Data Engineer with strong hands-on experience in Databricks-based lakehouse platforms. The role involves building and supporting scalable batch and real-time data pipelines using Spark, Kafka, and AWS.
This is a hands-on engineering role focused on production-grade data pipelines and modern cloud data architecture.
Required Skills & Qualifications
- Strong experience in Databricks (Jobs, Workflows, Notebooks)
- Strong Apache Spark (PySpark / Spark SQL)
- Delta Lake and lakehouse architecture
- Delta Live Tables (DLT) pipelines
- Kafka streaming data experience
- Advanced SQL
- AWS (S3, IAM, etc.)
- ETL/ELT pipeline development
- Production data engineering experience