What are the responsibilities and job description for the Lead Data Engineer position at Argyll Infotech Enterprise Pvt Ltd?
Hello Everyone,
No GC,
Role: Senior Lead Data Engineer
Location: San Jose, CA 95125 Hybrid(Locals)
Key: Hadoop, Spark SQL strong
No GC,
Role: Senior Lead Data Engineer
Location: San Jose, CA 95125 Hybrid(Locals)
Key: Hadoop, Spark SQL strong
- Key Responsibilities:
- Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL
- - Manage and optimize the infrastructure ensuring reliability and performance
- -Design and implement data models and schemas for enterprise data warehouse and data marts
- -Write and maintain scripts in Python for various applications
- -Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases
- - Work closely with analysts and product managers to understand data requirements and deliver high-quality data
- -Provide technical guidance and mentorship to other team members fostering best practices in data engineering
- -Monitor system performance, identify bottlenecks, implement solutions to ensure high data availability
- -High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing
- -Good knowledge in Python for data manipulation and scripting
- -Mastery of advanced SQL and deep experience working with relational databases and data warehouses
- -Hands-on experience with workflow management tools like Apache Airflow
- -Familiarity with Unix shell scripting for automation and system management
Salary : $60 - $65