What are the responsibilities and job description for the Senior Data Engineer position at Tek Leaders Inc?
Senior Data Engineer
data Engineer that is good with Pyspark and SQL. On prem work not cloud but GCP is a nice to have
Hybrid twice a week in Hoboken, NJ
Needs : 12 IT Experience
Pyspark
Python
GCP Nice to have
Good CLients
Any work authorization
6 month contract
You can subvend
Job Description : About Team :
Everyone has data, but the sheer volume of data at Walmart can be limitless. In the Data Engineering team, we help Walmart manage this data by building pipelines and data lakes to prepare big data for analysis and unlocking actionable insights in real-time. We also use cross-departmental data and machine learning to build a holistic view of true profitability, saving millions of dollars across item categories and geographies while assisting our leadership in making better decisions faster.
What you'll do :
- Collaborate with cross-functional teams to understand data requirements and design data solutions that meet business needs
- Develop and maintain data pipelines and ETL processes using Spark and Scala
- Design, build, and optimize data models and data architecture for efficient data processing and storage
- Implement data integration and data transformation workflows to ensure data quality and consistency
- Monitor and troubleshoot data pipelines to ensure data availability and reliability
- Conduct performance tuning and optimization of data processing systems for improved efficiency and scalability
- Work closely with data scientists and analysts to provide them with the necessary data sets and tools for analysis and reporting
- Stay up-to-date with the latest industry trends and technologies in data engineering and apply them to enhance the data infrastructure
What you'll bring :
