What are the responsibilities and job description for the AWS Databricks Consultant position at J-RAM IT Consulting Inc.?
AWS Databricks Consultant
Introduction:
We are looking for an experienced AWS Databricks Consultant to join our team in Dallas, TX. The ideal candidate will collaborate as part of a development team to design and enhance large scale applications using Python, Spark, and Pyspark. Real-time experience on Databricks is a must for this role. If you have a strong background in data analysis, data mapping, and data integration using AWS technologies, we want to hear from you.
Responsibilities:
- Evaluate and plan software designs, test results, and technical manuals using AWS.
- Confer with business units and development staff to understand both business and technical requirements for producing technical solutions.
- Create and review technical and user-focused documentation for data solutions.
- Extend and enhance the business Data Lake.
- Create or implement solutions for metadata management.
- Design and execute strategies for real-time data analysis and decision-making.
- Build robust data processing pipelines using AWS Services and integrate with multiple data sources.
- Translate client user requirements into data flows and data mapping.
- Analyze and determine data integration needs following Agile practices.
Requirements:
Required Skills:
- 4 years of experience designing and developing Data Pipelines for Data Ingestion or Transformation using Scala or Python.
- 4 years of experience with Python, Spark, and Pyspark.
- 3 years of experience working on AWS technologies.
- Experience designing, building, and deploying production-level data pipelines using tools from AWS Glue, Lambda, Kinesis, Aurora, and Redshift.
- Experience with Spark programming (Pyspark or Scala).
- Hands-on experience with AWS components like EMR, S3, Redshift, Lambda, API Gateway, and Kinesis in production environments.
- Strong analytical skills and advanced SQL knowledge, indexing, query optimization techniques.
- Experience using ETL tools for data ingestion.
- Experience with Change Data Capture (CDC) technologies and relational databases such as MS SQL, Oracle, and DB.
- Ability to translate data needs into detailed functional and technical designs for development, testing, and implementation.