What are the responsibilities and job description for the Data Engineer position at innovitusa?
We are looking for a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data processing systems. The ideal candidate will have experience in SQL, Python, ETL/ELT processes, data warehousing, and cloud-based data platforms.
Key ResponsibilitiesDesign, develop, and maintain robust data pipelines.
Build and optimize ETL/ELT workflows to integrate data from multiple sources.
Write complex SQL queries and optimize database performance.
Develop data processing solutions using Python, PySpark, or Spark.
Design and maintain data warehouses, data lakes, and data models.
Work with cloud platforms such as AWS, Azure, or Google Cloud.
Ensure data quality, accuracy, consistency, and security.
Monitor pipeline performance and troubleshoot data-related issues.
Collaborate with data analysts, data scientists, and business stakeholders.
Implement data governance, documentation, and best practices.
Strong proficiency in SQL and Python.
Experience with ETL/ELT tools and data integration.
Knowledge of Apache Spark and PySpark.
Experience with relational databases such as PostgreSQL, MySQL, or SQL Server.
Understanding of data warehousing concepts and dimensional modeling.
Experience with cloud platforms such as AWS, Azure, or GCP.
Familiarity with workflow orchestration tools such as Apache Airflow.
Knowledge of Git, version control, and CI/CD practices.
Strong analytical, problem-solving, and communication skills.
Bachelor's degree in Computer Science, Information Technology, or a related field.
Experience with Databricks, Snowflake, or Amazon Redshift.
Knowledge of Kafka and real-time data processing.
Experience with Docker, Kubernetes, or infrastructure as code.
Understanding of data lakehouse architecture and distributed computing.
Reliability and performance of data pipelines.
Data quality and accuracy.
Efficiency of data processing jobs.
Reduction in pipeline failures and processing time.
Timely delivery of data to business users.