What are the responsibilities and job description for the ETL / Data Engineer (BigQuery & Starburst) position at CogniSoft Technologies?
Key Responsibilities
- Design and develop scalable ETL/ELT pipelines for structured and semi-structured data
- Build and optimize data ingestion, transformation, and loading processes into BigQuery
- Develop federated query solutions using Starburst (Trino/Presto) across heterogeneous data sources
- Implement data modeling, schema design, and partitioning strategies for performance optimization
- Ensure data quality, validation, and governance across pipelines
- Collaborate with data science and AI teams to enable downstream analytics and Gen AI use cases
- Manage scheduling, orchestration, and monitoring of ETL workflows
Required Skills
- Strong experience in Google BigQuery (data modeling, querying, performance optimization, cost management)
- Hands-on experience with ETL/ELT pipeline development and large-scale data processing
- Experience with Starburst / Trino / Presto for distributed query processing
- Proficiency in Python, SQL, and Spark-based processing frameworks
- Experience with workflow orchestration tools (Airflow, ADF, etc.)
- Strong understanding of data warehousing concepts and data governance
Preferred
- Experience with cloud data platforms (Google Cloud Platform, AWS, Azure)
- Exposure to data lakehouse architectures and federated query models
- Understanding of supporting AI/ML and analytics workloads