What are the responsibilities and job description for the Big Data Lead with Databricks Experience (VISA INDEPENDENT CANDIDATES ONLY) position at SumasEdge Corporation?
Responsibilities:
- Design & Implement Data ingestion and Data lakes-based solutions using Big Data Technologies.
- The Tech Lead should be highly proficient in the use of Big Data / Open-Source Technologies and standard techniques of Data Integration, Data Manipulation.
- Should be able to design and develop cost efficient and performant data pipelines in the cloud platform
- Create data environment to support our data analytics, reporting and data science teams
- Experience with integration of data from multiple data sources
- Knowledge of various Data Pipeline techniques and frameworks
- Performance optimization - need to monitor the complete process and apply necessary infrastructure changes to speed up the query execution.
- Efficient data ingestion - Discovering patterns in data sets with data mining techniques and using different data ingestion APIs and inject data into the data lake as per need.
The Role offers:
- Great opportunities to learn various tools and technologies used in a sophisticated data architecture within the Business Intelligence and Analytics Data Services
- Gives an opportunity to showcase candidates strong analytical skills and problem-solving ability
- An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations
- Grow into a Technical architect role over a period
Essential Skills:
- 6 Years hands on knowledge on SQL as well as SQL/NoSQL databases
- Proficient in programming languages such as Python, PySpark, Scala and Java
- Experience with Spark , Databricks
- Working knowledge of XML, ETL, API and Web Services
- Experience with integration of data from multiple data sources
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
- Knowledge of various ETL techniques and frameworks, such as Flume
- Experience with various messaging systems, such as Kafka or RabbitMQ
- Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
- Working knowledge and experience in Big data services in one of the Cloud Provider will be good (AWS or Azure or Google Cloud Platform)
- Experience in leading offshore teams