What are the responsibilities and job description for the Data Engineer position at Avance Consulting?
Job Title: Spark and Scala Lead Developer
Location: Cupertino, CA, Austin TX
Duration: Full Time
Required Qualifications
- At least 4 years of experience with Information Technology
- Strong understanding of distributed computing principles and big data technologies
- Hands on experience working with Apache Spark, Scala, Spark SQL and Starburst
- Hands on experience with Big Data systems, building ETL pipelines, data processing, and analytics tools
- Understanding of data structures, algorithms & common methods in data transformation
- Familiar with the concepts of dimensional modeling
- Sound knowledge of one programming language - Python or Java
- Programming experience using tools such as Hadoop and Spark
- Strong proficiency in using query languages such as SQL, Hive and SparkSQL
- Experience in Scala Functional programming
Preferred Qualifications:
- Experience in Kafka would be a plus
- Knowledge of data serialization formats such as Parquet, Avro, or ORC
- Familiarity with data processing and transformation techniques
- Experience with data lakes, data warehouses, and ETL processes
- Good understanding of Agile software development frameworks
- Strong communication and Analytical skills
- Ability to work in teams in a diverse, multi-stakeholder environment comprising of Business and Technology teams
- Experience and desire to work in a global delivery environment