What are the responsibilities and job description for the Senior Data Engineer position at Software Guidance & Assistance, Inc. (SGA, Inc.)?
Software Guidance & Assistance, Inc., (SGA), is searching for an Senior Data Engineer for a FULL TIME assignment with one of our premier Data and Analytics client in their D.C office. This is a remote opportunity for candidates on the East Coast, preferably close to the DC Metro area.
Overview
Our client is seeking a Senior Data Engineer to design, build, and optimize large-scale distributed data systems in a high-performance, cost-conscious environment. This is a true senior-level individual contributor role focused on Scala-based Spark engineering, AWS EMR optimization, and distributed systems performance at multi-terabyte scale.This is not a generic "data engineer” role. We are looking for someone who deeply understands Spark internals, distributed resource management, and cost/performance tradeoffs — and who can clearly articulate the why behind their technical decisions.You will work on complex, high-volume data pipelines that directly power production systems. Success in this role requires ownership, a metrics-first mindset, and the ability to balance new feature development with rigorous operational excellence.
Key Responsibilities
This role is ideal for a senior engineer who:
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.
Overview
Our client is seeking a Senior Data Engineer to design, build, and optimize large-scale distributed data systems in a high-performance, cost-conscious environment. This is a true senior-level individual contributor role focused on Scala-based Spark engineering, AWS EMR optimization, and distributed systems performance at multi-terabyte scale.This is not a generic "data engineer” role. We are looking for someone who deeply understands Spark internals, distributed resource management, and cost/performance tradeoffs — and who can clearly articulate the why behind their technical decisions.You will work on complex, high-volume data pipelines that directly power production systems. Success in this role requires ownership, a metrics-first mindset, and the ability to balance new feature development with rigorous operational excellence.
Key Responsibilities
- Design and develop high-performance data pipelines using Apache Spark with Scala (Scala-only; no PySpark environments).
- Optimize large-scale, multi-terabyte daily workloads for both performance and cost efficiency.
- Manage and tune AWS EMR environments, including fleet management and instance optimization.
- Build and orchestrate workflows using AWS Step Functions.
- Write efficient Snowflake/Snowpark procedures across Scala, SQL, and Python as required.
- Diagnose and resolve complex distributed system issues, including data skew and resource contention.
- Implement observability and monitoring from day one (metrics, alerting, performance baselines).
- Continuously improve existing pipelines while building new features and data capabilities.
- Proactively identify inefficiencies and drive measurable optimization improvements.
- 5 years hands-on Apache Spark experience using Scala exclusively (no PySpark backgrounds).
- Proven experience optimizing large-scale Spark jobs at multi-terabyte scale.
- Strong AWS EMR experience, including cluster sizing, fleet management, and cost tuning.
- Proficiency with AWS Step Functions.
- Deep understanding of distributed computing principles and resource management.
- Demonstrated experience mitigating data skew and tuning performance bottlenecks.
- Experience debugging complex distributed systems issues in production environments.
- Comfort working across Scala, SQL, and Python when needed.
- Experience with probabilistic data structures for high-cardinality processing.
- Advanced troubleshooting and root cause analysis skills.
- Metrics-first mindset — measures before and after optimization.
- Strong understanding of cost/performance tradeoffs in cloud environments.
- Ability to interpret business needs beyond written requirements.
- Clear communicator who escalates risks and blockers early.
- Ownership mentality — sees inefficiencies as opportunities to improve.
- Experience designing large-scale, observable, and maintainable data platforms.
- Track record of balancing innovation with operational excellence.
- Experience mentoring other engineers and contributing to technical standards.
- Prior experience in environments where efficiency, scale, and cost control are mission-critical.
This role is ideal for a senior engineer who:
- Understands Spark internals — not just APIs.
- Has operated EMR clusters in production and tuned them for cost and performance.
- Thrives in ambiguity and takes initiative.
- Can explain tradeoffs clearly to both technical and non-technical stakeholders.
- Believes "it runs” is not the same as "it runs efficiently.”
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.