Data Engineer

Job Title: Data Engineer (8 Years)

Location: Atlanta, GA (Onsite)

Employment: Full Time/ W2 (NO C2C)

Job Summary: We are seeking an experienced Senior Data Engineer with 8 years of hands-on experience designing, building, and optimizing scalable, high-performance data platforms. The ideal candidate will have strong expertise in data ingestion, transformation, data warehousing, cloud platforms, big-data technologies, and analytics enablement. This role involves close collaboration with Data Architects, Analytics, Data Science, Product, and Engineering teams to deliver reliable, secure, and analytics-ready data solutions.

Key Responsibilities:

• Design, develop, and maintain end-to-end data pipelines for structured and semi-structured data using batch and real-time processing frameworks.
• Build and optimize ETL/ELT pipelines using cloud-native and big-data tools to ingest data from databases, APIs, files, event streams, and third-party sources.
• Develop data transformation logic using SQL, Python, PySpark, and Spark SQL to support analytics, reporting, and data science workloads.
• Implement and manage cloud-based data platforms leveraging Azure, AWS, or GCP, including Data Lakes, Lakehouse architectures, and Data Warehouses.
• Design and optimize Data Lake (Bronze/Silver/Gold) layers, Delta/Parquet formats, partitioning strategies, and performance tuning techniques.
• Build and maintain Data Warehouses and analytical models (star/snowflake schemas) to support BI, dashboards, and regulatory reporting.
• Work with streaming data technologies to support near real-time ingestion and processing using Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Ensure data quality, validation, reconciliation, and lineage, implementing robust error handling, logging, and monitoring frameworks.
• Collaborate with Data Analysts, BI teams, and Data Scientists to deliver analytics-ready datasets and curated views.
• Implement security, governance, and compliance controls, including RBAC, encryption, masking, auditing, and metadata management.
• Support CI/CD pipelines, version control, and automated deployments for data engineering solutions.
• Participate in Agile/Scrum ceremonies, providing accurate estimates, documentation, and continuous improvement.
• Troubleshoot and resolve performance bottlenecks, data issues, and production incidents.

Required Skills:

• Strong proficiency in SQL, Python, PySpark, and Spark for large-scale data processing.
• Hands-on experience with Cloud Platforms: Azure (ADF, ADLS, Synapse, Databricks, Fabric), AWS (S3, Glue, Redshift, EMR), or GCP (BigQuery, Dataflow, Dataproc).
• Solid experience with Data Warehousing concepts, dimensional modeling, and analytical data design.
• Experience building ETL/ELT pipelines using tools such as ADF, SSIS, Glue, Airflow, Informatica, or dbt.
• Knowledge of Big Data ecosystems, including HDFS, Hive, Spark, Kafka, and distributed computing concepts.
• Familiarity with BI and reporting tools such as Power BI, Tableau, Looker, or SSRS.
• Strong understanding of data quality, governance, metadata, and master data management.
• Experience working in Agile environments with tools like JIRA, Confluence, and Git.

Preferred Qualifications:

• Experience with Lakehouse architectures and Delta Lake.
• Exposure to Microsoft Fabric, Synapse Analytics, and modern analytics platforms.
• Knowledge of DevOps and CI/CD for data platforms (Azure DevOps, GitHub Actions, Jenkins).
• Experience with containerization and orchestration (Docker, Kubernetes).
• Background in regulated industries such as Banking, Healthcare, or Insurance.
• Certifications in Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer are a plus.