What are the responsibilities and job description for the Data Engineer position at Redolent, Inc?
One of our direct client is urgently looking for a Data Engineer @ Dallas TX
TITLE: Data Engineer
LOCATION: Dallas TX
Duration: 6 to 12 Months.
Description:
Design, develop and build database to power Big Data analytical systems.
• Design data integration pipeline architecture and ensure successful creation of the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Spark, SQL, HQL and other technologies.
• Build robust and scalable applications using SQL, Scala/Python and Spark.
• Create real time data streaming and processing using Kafka and/or Spark streaming.
• Work on creating data ingestion processes to maintain Global Data lake on Google cloud or Azure
• Engage with architects and senior technical leads to create and enhance complex software components.
• Design, configure and implement systems that can scale to process terabytes of data between heterogeneous systems on premise and cloud.
• Work with business customers, product managers and engineers to design feature-based solutions and implement them in an agile fashion.
• Develop proof-of-concept prototype with fast iteration and experimentation.
• Develop and maintain design documentation, test cases, performance and monitoring and performance evaluation using Git, Crontab, Putty, Jenkins, Maven, Confluence, ETL, Automic, Zookeeper, Cluster Manager
• Perform continuous integration and deployment using Jenkins and Git