What are the responsibilities and job description for the Pyspark and Databricks Engineer position at Avance Consulting?
Job Title : Pyspark and Databricks Engineer
Location : Pittsburgh, Pennsylvania ( Onsite )
Job Type : Full-time
Required Skill and Experience
Strong programming experience in Python
Hands-on expertise with PySpark
Proven experience with Databricks (jobs, notebooks, Delta Lake)
Experience working with Kafka for real-time data ingestion
Strong SQL and data processing fundamentals
Experience with distributed data processing and big data ecosystems
Preferred Skill and Experience
Cloud platform experience (AWS/Azure/GCP)
Experience with Delta Lake or Lakehouse architecture
Knowledge of CI/CD and data orchestration tools