What are the responsibilities and job description for the Data Architect (Azure Databricks) position at TestingXperts?
Role: Data Architect (Azure Databricks)
Location: Plano, TX (3 days a week)
Fulltime
Responsibilities
- Manage end to end delivery by Investigating problem areas, working cross-functionally with product manager & other stakeholders.
- Follow the Agile development methodology; think strategically and execute methodically.
- Develop and manage capacity and growth projection forecasts of the environment within budgets.
- Create and maintain optimal data pipeline architecture, assemble large, complex data sets that meet functional / non-functional business requirements.
- Drive optimization, testing and tooling to improve quality of solutions.
- Manage teams that build and operate high volume distributed systems in a SaaS environment.
- Great at devising efficient processes that increase velocity and quality.
Desired Experience/ Skills:
- 8 years of relevant experience
- At least 5 years of experience building and leading highly complex technical engineering teams.
- Strong hands-on experience in Databricks
- Experience managing distributed teams preferred.
- Strong technical experience in large distributed systems, Data Warehousing, Data Lake at scale Project management skills: financial/budget management, scheduling and resource management experience with medium and large-scale projects
- Comfortable working with ambiguity and multiple stakeholders.
- Experience in medallion architecture, optimization, Ingestion, Quality.
- Should have worked on Unity Catalog and understands its architecture and how to leverage its Data Governance and Observability capabilities.
- Comfortable working cross functionality with product management and directly with customers; ability to deeply understand product and customer personas.
- Architecture Design Experience for Cloud and Non-cloud platforms
- Expertise on Azure Cloud platform
- Knowledge on orchestrating workloads on cloud
- Ability to set and lead the technical vision while balancing business drivers
- Strong experience with PySpark, Python programming
- Proficiency with APIs, containerization and orchestration is a plus
Qualifications:
- Bachelor's and/or master’s degree