What are the responsibilities and job description for the Data Platform Engineer position at TalentBridge?
Title: Data Platform Engineer
Location: San Jose, CA 95110 (Hybrid)
Duration: 06 Months (possible extension)
Job Description:
- Setup and maintain production-scale Databricks environments on public cloud platforms such as Microsoft Azure and AWS
- Handle production-scale data storage platforms like ADLS and S3 for multiple tenant teams within our Data Platform
- Coordinate production-scale microservices that support the daily operations of our data platform, including job scheduling, security, financial, and administrative services
- Maintain enterprise-scale SQL Server and SSAS environments
- Provide triage and mentorship to the team on various support issues raised by our tenants
- Develop tools and automation solutions for configuration management, service deployments, monitoring, and alerting
- Ensure security and privacy compliance, implementing Adobe Security & Compliance solutions to secure data in our data lake
- Explore GenAI technologies to integrate technical improvements and improve user experience
- Collaborate with various 3rd party vendors to resolve issues, conduct proof of concepts, and improve our product
Cloud Infrastructure Administration and Automation:
- AWS
- Azure
- ADLS Gen2
- S3
- MongoDB
- Vector Databases
- AI Agent Development and Maintenance:
- Langchain
- Databricks
- SQL Server & SSAS
- Airflow
- Azure Kubernetes Service or Elastic Kubernetes Service
- DBX System Tables
- Prometheus
- Splunk
- MLFlow
- Linux Servers
- Kubernetes Environment
- Jira
- Service Now