What are the responsibilities and job description for the Senior Data Engineer AWS position at LiveMindz?
- Brings 12 years of data engineering experience spanning production ETL/ELT pipelines, Big Data processing, cloud migrations, data lakes, and analytics platforms.
- Builds and supports AWS Glue ETL pipelines using PySpark/Python, including ingestion, transformations, schema validation, data-quality checks, and curated data layers.
- Develops event-driven and near-real-time architectures using AWS Lambda, Eventbridge, Step Functions, Kafka/MSK, and SNS.
- Has hands-on experience with Glue Catalog and Glue Crawlers for metadata discovery, schema registration, governance, and downstream querying through Athena.
- Uses CloudWatch logs, metrics, monitoring, and alerts to troubleshoot Glue, Lambda, and Step Functions and maintain pipeline reliability and SLA compliance.
- Has strong Python, PySpark, SQL, PostgreSQL, Redshift, Hive, Snowflake, and Databricks experience across large structured and semi-structured datasets.
- Implements CDC, incremental processing, schema enforcement, error handling, retry logic, data-quality validation, and scalable batch/streaming pipelines using PySpark, Delta Lake, and Delta Live Tables.
- Builds orchestration solutions using Step Functions, Airflow, Lambda, dbt, Control-M, and AWS services, managing workflow dependencies, scheduling, retries, and cross-service execution.
- Automates infrastructure with Terraform and Cloud Formation and supports CI/CD through Git, Jenkins, AWS Code Pipeline, and Docker.
- Has extensive cloud-migration experience, including moving large on-premises Hadoop/Teradata workloads to AWS and designing scalable S3-based data lake architectures.
- Brings hands-on performance tuning experience across Spark/PySpark jobs, SQL queries, Hive workloads, Redshift clusters, partitioning, and large-scale data pipelines.
- Works effectively with business, BI, analytics, and downstream data consumers and has supported Tableau, Snowflake, Redshift, and Athena-based analytics environments.
CERTIFICATIONS
- AWS Certified Developer – Associate