Data Engineer - Must Have Aws experience
Fulltime/Permanent - 4 Positions
2 for Memphis, TN and 2 for St. Louis, MO - Onsite / Hybrid

The candidates are being interviewed by high-level AWS Consultants who were responsible for client's AWS migration and want to make sure that whoever takes over this role is fully qualified.

They need someone who is well versed and highly experienced using AWS Glue because it is how they are going to integrate data (and prepare and move the data) from the different sources (think of Glue as a cloud ETL tool like SSIS but for AWS, not SQL Server). However, there are limitations to Glue which is why they need to have experience with other tools like AWS Lake Formation and AWS Lambda.

This is the bullet in the job description:

• Strong understanding of AWS services related to data engineering, such as Amazon S3, Amazon Redshift, Amazon Aurora Postgres, AWS Glue, AWS Lambda, AWS Step Function, AWS Lake Formation, Amazon Data Zone, Amazon Kinesis, MSK and Amazon EMR.

Job Summary: The data engineer's main responsibility is to design, develop, and maintain data pipelines and infrastructure on the AWS platform. Working with large volumes of data, ensuring its quality, reliability, and accessibility. Tasks may include data ingestion, transformation, and storage, data sharing and consumption as well as implementing data security and privacy measures. This role is crucial in enabling efficient and effective data-driven decision-making within the company.
Essential Duties and Responsibilities

1. Build and maintain scalable and reliable data pipelines, ensuring the smooth flow of data from various sources to the desired destinations in AWS cloud environment.
2. Work closely with stakeholders to understand their data requirements and designing data solutions that meet their needs. This includes understanding data models/schemas, and implementing ETL (Extract, Transform, and Load) processes to transform raw data into a usable format in destination.
3. Responsible for monitoring and optimizing the performance of data pipelines, troubleshooting any issues that arise, and ensuring data quality and integrity.

Qualifications & Technology

1. Proficient in programming languages such as Python, as well as SQL for database querying and manipulation.
2. Strong understanding of AWS services related to data engineering, such as Amazon S3, Amazon Redshift, Amazon Aurora Postgres, AWS Glue, AWS Lambda, AWS Step Function, AWS Lake Formation, Amazon Data Zone, Amazon Kinesis, MSK and Amazon EMR.
3. Knowledge of database design principles and experience with database management systems
4. Experience with data storage technologies like relational databases (e.g., SQL Server, PostgreSQL) and distributed storage systems (e.g., PySpark) is essential.
5. Understanding of Extract, Transform, Load (ETL) processes and experience with ETL tools like AWS Glue and SQL Server Integration Services is important. You should be skilled at integrating disparate data sources and ensuring data quality and consistency.
6. Understanding and experience with orchestration tools like Apache Airflow, AWS Glue Workflows, AWS Step Functions and notification services.
7. Familiarity to IAC such as Terraform, git and DevOps pipelines
8. Awareness of data governance practices, data privacy regulations, and security protocols is crucial. Experience implementing data security measures and ensuring compliance with relevant standards is desirable.
9. Strong analytical thinking and problem-solving abilities are essential to identify and resolve data-related issues effectively. You should be able to analyze complex data sets, identify patterns and derive actionable insights.

Education and Experience

1. Minimum Required: Bachelor's Degree in Computer Science or related field or equivalent experience
2. Minimum Required: 8 to 10 years' of post Bachelor's progressive experience in Data Engineering