Senior Data Engineer / Databricks Specialist

Satsyil Corp is seeking a highly skilled and motivated Senior Data Engineer / Databricks Specialist to join our team. In this role, you will play a crucial part in designing, developing, and optimizing high-performance Spark applications and managing enterprise data infrastructure within AWS Databricks. Utilizing PySpark, Python, SQL, and Delta Lake, you will architect robust ETL/ELT pipelines, maintain medallion architecture, and establish efficient big data solutions supporting mission-critical public sector and enterprise environments.

·       Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL to support data processing and analytics needs.

·       Design, build, and maintain robust ETL/ELT pipelines and implement Medallion Architecture (Bronze, Silver, Gold layers) ensuring data integrity, completeness, and lineage.

·       Administer Databricks Workspaces, including cluster provisioning, job scheduling, access controls, Unity Catalog, and notebook management.

·       Manage AWS cloud infrastructure assets including S3, Lambda, IAM, Aurora, Glue, and provision environments using Terraform Infrastructure-as-Code (IaC).

·       Perform query optimization, performance tuning, troubleshooting, and debugging for complex Spark jobs and large-scale data solutions.

·       Merge, consolidate, and process structured, semi-structured, and unstructured data sets from diverse API endpoints and databases.

·       Embed security gates and compliance controls within DevSecOps CI/CD pipelines (Jenkins, GitLab CI) to support FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.

·       Implement rigorous data quality controls, anomaly detection, validation rules, and automated lineage tracking across all pipelines.

·       Collaborate within cross-functional Agile/Scrum teams in multi-contractor federal environments, providing technical leadership and production on-call support.

Required Experience & Technical Skills:

·       Experience: 3 years of dedicated Data Engineering / Platform Administration experience, and hands-on experience in Databricks.

·       Core Big Data Stack: Production proficiency with PySpark, Spark SQL, Apache Spark, and Delta Lake.

·       Cloud Ecosystem: Strong experience in AWS Cloud services including S3, IAM, Lambda, Glue, and Aurora.

·       Automation & DevOps: Production experience with Terraform (IaC) and CI/CD tools (Jenkins, GitLab CI, or equivalent).

·       Languages & Frameworks: Deep proficiency in Python and SQL; working knowledge of REST APIs for ingestion.

·       Clearance & Citizenship: U.S. Citizenship required; must have the ability to obtain and maintain a Public Trust clearance.

·       Methodology & Education: Hands-on experience in Agile/Scrum delivery environments. Bachelor's degree in CS, IT, Data Science, or related engineering discipline.

Preferred Qualifications & Nice-to-Haves:

·       Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and AI/ML pipeline support (MLflow, Model Registry).

·       Experience in Federal Government contracting and compliance frameworks (FISMA / FedRAMP environments).

·       Active security clearance (Public Trust or higher).

Candidates possessing any of the following certifications will be prioritized:

·       Databricks Certified Data Engineer Associate

·       Databricks Certified Data Engineer Professional

·       Databricks Certified Machine Learning Associate

·       AWS Certified Solutions Architect – Associate or Professional

·       AWS Certified Data Analytics – Specialty