What are the responsibilities and job description for the Senior Data Engineer - Databricks & GenAI || Warren, MI OR Austin, TX - 4 days/week position at HYR GLOBAL SOURCE INC?
Job Title: Senior Data Engineer – Databricks & GenAI
Location: Warren, MI OR Austin, TX - 4 days/week
Employment Type: 6 months contract to fulltime hire with client.
Interview Process: One technical video interview, followed by an offer decision.
Experience: 5 years in Data Engineering
Location: Warren, MI OR Austin, TX - 4 days/week
Employment Type: 6 months contract to fulltime hire with client.
Interview Process: One technical video interview, followed by an offer decision.
Experience: 5 years in Data Engineering
Job Summary: We are looking for a Senior Data Engineer with strong hands-on experience in Databricks, PySpark, Python, and SQL. The candidate will build and maintain data pipelines, organize large amounts of data, and prepare data for AI and Generative AI applications.
Candidates must have experience working with Databricks on Azure or GCP, with strong practical knowledge of data engineering and data processing.
Key Responsibilities
- Build and maintain data pipelines using Databricks, PySpark, Python, and SQL.
- Process, clean, and organize data from different sources.
- Work with Delta Lake and Unity Catalog to manage and organize data.
- Build reliable data pipelines with proper testing, monitoring, and data quality checks.
- Prepare data for Machine Learning and Generative AI applications, including embeddings and RAG-based solutions.
- Work with cloud services, especially Azure data storage and security tools.
- Collaborate with data, application, and AI teams.
- Write technical documentation and participate in code reviews.
Required Skills
- 5 years of experience in Data Engineering.
- Strong hands-on experience with Databricks.
- Strong experience with PySpark, Python, and SQL.
- Experience with Databricks on Azure or GCP.
- Hands-on experience with Delta Lake and Unity Catalog.
- Experience building and managing production data pipelines.
- Knowledge of Git, code testing, and data quality checks.
- Good communication and problem-solving skills.
Preferred Skills
- Experience with Generative AI, RAG, embeddings, or vector search.
- Experience preparing data for Machine Learning applications.
- Knowledge of Azure services such as ADLS Gen2 and Key Vault.
- Experience with real-time data processing or streaming.
- Exposure to MLOps, model serving, or feature stores.
- Experience with manufacturing or automotive data is a plus, but not mandatory.
Important Requirements
- Must be willing to work onsite 4 days per week in either Warren, MI or Austin, TX.
- Must have strong, hands-on Databricks, PySpark, and SQL experience.
- Must be comfortable converting to a full-time employee after approximately 6 months on contract.
- Candidates should be comfortable working with different teams and adapting to new technical environments.
#LI-SS3