What are the responsibilities and job description for the Databricks Senior Data Engineer -Pyspark position at SmartIT Frame?
Hiring: Senior Data Engineer – Databricks/PySpark
Location: New York, NY – Onsite
Requirements:
-
7 years of Data Engineering experience
-
4 years hands-on with Databricks & Spark/PySpark
-
Strong Python, SQL, Delta Lake
-
Experience with AWS (S3, Glue) and Google Cloud Platform (BigQuery, GCS)
-
Strong ETL/ELT, Spark optimization & data pipeline development
-
Experience with CI/CD, Git, pytest
-
Excellent communication and technical documentation skills
-
Consulting/project delivery experience
Nice to Have: Kafka, Spark Streaming, DLT, Unity Catalog, Docker/Kubernetes.