What are the responsibilities and job description for the Hadoop Platform support position at ApTask?
Job Title: Hadoop Platform support
Location: Charlotte, NC
Mphasis/Wells Fargo
Job Summary: We are seeking a skilled Data Engineer with a strong focus on Hadoop Distributed File System (HDFS) to join our team. The ideal candidate will be responsible for the support and maintenance of our Hadoop platform, ensuring optimal performance and reliability. This role requires a hands-on approach to platform support (L3 Support) and a solid understanding of Google Cloud Platform (Google Cloud Platform) to effectively manage data workflows and storage solutions.
Responsibilities:
- Provide L3 support for the Hadoop platform, troubleshooting and resolving issues to ensure high availability and performance.
- Manage and maintain the Hadoop Distributed File System (HDFS), ensuring data integrity and security.
- Collaborate with data scientists and analysts to understand data requirements and optimize data pipelines.
- Implement and monitor data ingestion processes, ensuring efficient data flow from various sources into the Hadoop ecosystem.
- Work with Google Cloud Platform services to integrate cloud-based solutions with on-premises Hadoop infrastructure.
- Develop and maintain documentation for data engineering processes, including data models, workflows, and system configurations.
- Participate in capacity planning and performance tuning of the Hadoop environment.
- Stay updated with the latest industry trends and technologies related to data engineering and Hadoop.
Mandatory Skills:
- Strong knowledge of Hadoop Distributed File System (HDFS) and its architecture.
- Hands-on experience with L3 support and maintenance of Hadoop platforms.
- Proficiency in managing and optimizing data workflows within the Hadoop ecosystem.
- Familiarity with Google Cloud Platform (Google Cloud Platform) and its data services.
- Experience with data ingestion tools and techniques.
Preferred Skills:
- Knowledge of additional big data technologies such as Apache Spark, Hive, or Pig.
- Experience with data modeling and ETL processes.
- Familiarity with programming languages such as Python, Java, or Scala.
- Understanding of data governance and security best practices.
- Experience with containerization technologies like Docker or Kubernetes.
Qualifications:
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- Relevant certifications in Hadoop or cloud technologies are a plus.
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration abilities.