What are the responsibilities and job description for the Senior Data Engineer position at Quantum Business Systems, Inc?
Key Responsibilities
- Build and maintain ETL/ELT data pipelines using PySpark and Databricks.
- Work with distributed datasets, ensuring efficiency, scalability, and accuracy.
- Develop and optimize workflows for data ingestion, transformation, and warehousing.
- Write high-quality Python code for data engineering use cases.
- Collaborate with data science and analytics teams to translate requirements into solutions.
- Troubleshoot issues, optimize queries, and manage performance within AWS cloud environments.
- Support CI/CD and automation workflows for reliability and repeatability.
Required Skills & Experience
- Strong hands-on experience in Python and PySpark for large-scale data engineering.
- Proven expertise in using Databricks for data transformation and analytics.
- Solid working knowledge of SQL and cloud data platforms (e.g., Snowflake).
- Experience with major cloud providers (AWS preferred; Azure/GCP considered).
- Good understanding of data modelling, data warehouses, and data lakes.
- Exposure to orchestration tools like Airflow, AWS Glue, or Azure Data Factory.
- Familiarity with DevOps practices, CI/CD, and version control systems.
- Strong problem-solving and debugging skills in distributed data environments.
Salary : $60 - $70