What are the responsibilities and job description for the Databricks Data Engineer - Houston (TX)- Onsite Day 1 position at K Anand Corporation?
Role- Databricks Data Engineer
Work Location: Houston (TX)- Onsite Day 1
Skills-
Data Engineer
We are seeking an Databricks Data Engineer who can play a crucial role in our Fintech data lake project.
What You Bring
- 4 years of experience working in data warehousing systems
- 2 strong hands-on programming expertise in Databricks landscape, including SparkSQL, Workflows for data processing and pipeline development
- 2 strong hands-on data transformation/ETL skills using Spark SQL, Pyspark, Unity Catalog working in Databricks Medallion architecture
- 1 yrs work experience in one of cloud platforms: Azure, AWS or Google Cloud Platform
- Good exposure to Git version control, and CI/CD best practices
- Experience in developing data ingestion pipelines from ERP systems (Oracle Fusion preferably) to a Databricks environment, using Fivetran or any alternative data connectors is a plus
- Experience in a fast-paced, ever-changing and growing environment
- Understanding of metadata management, data lineage, and data glossaries is a plus.
What You'll Do
- Involve in design and development of enterprise data solutions in Databricks, from ideation to deployment, ensuring robustness and scalability.
- Work with Sr. Data Engineer to build, and maintain robust and scalable data pipeline architectures on Databricks using PySpark and SQL
- Assemble and process large, complex ERP datasets to meet diverse functional and non-functional requirements.
- Involve in continuous optimization efforts, implementing testing and tooling techniques to enhance data solution quality
- Focus on improving performance, reliability, and maintainability of data pipelines.
- Implement and maintain PySpark and databricks SQL workflows for querying and analyzing large datasets
Qualifications
- Bachelors Degree in Computer Science, Engineering, Statistics, Finance or equivalent experience
- Good communication skills