What are the responsibilities and job description for the Data Engineer with Gen AI position at Skywaves MP LLC?
Key Responsibilities
- Design, build, and maintain scalable data pipelines using modern data engineering tools.
- Develop and manage data transformation workflows using dbt and Dagster.
- Build robust data models and pipelines supporting analytics and AI workloads.
- Develop backend services and APIs using Python and FastAPI to enable data access for AI applications.
- Work with PostgreSQL and SQL to design efficient schemas and optimize queries.
- Process and analyze data using Pandas and Python-based data frameworks.
- Integrate structured and unstructured data pipelines to support Generative AI applications.
- Collaborate with AI/ML engineers to enable LLM-powered applications and data pipelines.
- Implement data quality, monitoring, and performance optimization practices.
- Participate in architecture design discussions and contribute to scalable data platform design.
Required Skills
- Strong programming experience in Python.
- Experience working with Dagster for orchestration.
- Hands-on experience with dbt for data transformation and modeling.
- Advanced SQL skills and experience with PostgreSQL.
- Experience working with Pandas for data processing.
- Experience building APIs using FastAPI.
- Solid understanding of data pipeline design and ETL/ELT processes.
- Familiarity with building data infrastructure for AI/ML or GenAI use cases.
Good to Have
- Experience designing Star Schema / Dimensional Data Models.
- Familiarity with Medallion Architecture (Bronze, Silver, Gold layers).
- Experience with Generative AI frameworks or LLM integrations.
- Knowledge of RAG pipelines, vector databases, or embedding workflows.
Exposure to cloud platforms such as AWS / Google Cloud Platform / Azure.