What are the responsibilities and job description for the Data Engineer position at Tekcel8?
Job Description
We are seeking a highly skilled Data Engineer with 10 years of experience in designing, building, and optimizing scalable data pipelines and data platforms. The ideal candidate will have strong expertise in cloud technologies, big data frameworks, ETL processes, and data warehousing solutions.
Required Skills & Qualifications
- 10 years of experience in Data Engineering and Data Integration.
- Strong proficiency in Python, SQL, and Spark.
- Extensive experience building and maintaining ETL/ELT pipelines.
- Hands-on experience with Apache Spark, Databricks, Hadoop, or similar big data technologies.
- Strong knowledge of cloud platforms such as AWS, Azure, or Google Cloud Platform.
- Experience with cloud-native data services such as:
- AWS: S3, Glue, Redshift, Lambda, EMR
- Azure: Data Factory, Synapse Analytics, Data Lake Storage
- Google Cloud Platform: BigQuery, Dataflow, Cloud Storage
- Experience with data warehousing solutions including Snowflake, Redshift, Synapse, or BigQuery.
- Strong SQL skills with expertise in query optimization and performance tuning.
- Experience with data modeling, schema design, and data governance practices.
- Hands-on experience with orchestration tools such as Airflow, Azure Data Factory, or similar.
- Familiarity with CI/CD pipelines, Git, and DevOps practices.
- Experience working in Agile/Scrum environments.
- Strong analytical, problem-solving, and communication skills.
Preferred Qualifications
- Experience with real-time streaming technologies such as Kafka, Kinesis, or Event Hubs.
- Knowledge of Data Lake and Lakehouse architectures.
- Experience with Terraform or Infrastructure as Code (IaC).
- Exposure to machine learning data pipelines and MLOps concepts.
- Cloud certifications (AWS, Azure, or Google Cloud Platform) are highly preferred.
Responsibilities
- Design, develop, and maintain scalable data pipelines and data integration solutions.
- Build and optimize data ingestion, transformation, and processing frameworks.
- Develop and maintain enterprise data warehouses and data lakes.
- Collaborate with data analysts, data scientists, architects, and business stakeholders.
- Ensure data quality, integrity, security, and governance standards are met.
- Monitor and troubleshoot data pipeline performance issues.
- Implement best practices for automation, monitoring, and scalability.
- Participate in architecture discussions and contribute to strategic data initiatives.