We are currently seeking a Mid-Level Data Ops Engineer for a full time, permanent (W2 only) role. This will be working onsite 100% at the Pentagon. Candidates must have an active TS/SCI clearance. Role Description & Requirements
Our data engineers support data collection, ingestion, validation, and loading of optimized data in the appropriate data stores. They work on a team made up of analyst(s), developer(s), data scientist(s), and a product lead and everyone on the team collaborates in support of a specific the mission. Working directly with the analyst(s) and the product lead, the data engineer identifies and implements solutions for the data requirements, including building pipelines to collect data from disparate, external sources, implementing rules to validate that expected data is received, cleansed, transformed, massaged and in an optimized output format for the data store. The Data Engineer performs validation and analytics corresponding with client requirements and evolves solutions through automation, optimizing performance with minimal human involvement. As pipelines are executed, the data engineer monitors their status, performance, and troubleshoots issues while working on improvements to ensure the solution is the very best version to address the customer need. As a Mid-Level Data Ops Engineer, this role focuses specifically on the development and maintenance of scalable data stores that supply big data in forms needed for business analysis. The best athlete candidate for this position will be able to apply advanced consulting skills, extensive technical expertise and has full industry knowledge to develop innovative solutions to complex problems. This candidate is able to work without considerable direction and may mentor or supervise other team members. What we’re looking for: • Someone with a solid background developing solutions for high volume, low latency applications and can operate in a fast paced, highly collaborative environment.
• A candidate with distributed computer understanding and experience with SQL, Spark, ETL. • A person who appreciates the opportunity to be independent, creative and challenged. • An individual with a curious mind, passionate about solving problems quickly and bringing innovative ideas to the table. Basic Qualifications:
• 2 years of experience with Python and SQL
• Experience with connecting to other data systems through APIs, Web Services, SFTP, etc.
• Knowledge of the software development life cycle, testing, and version control
• Experience working with structured, unstructured and/or semi-structured data
• Experience working with data science projects and integrating AI workflows Additional Qualifications:
• Experience with event-driven micro-service architectures
• Experience with distributed programming languages and paradigms such as Spark, Scala, etc.
• Experience with data engineering tools such as Databricks, ElasticSearch, Apache NiFi, StreamSets, etc.
• Experience with data lakes, data warehouses, and/or data lake houses
• Experience with different database paradigms including but not limited to relational databases (Deltalake, SQL/PostgreSQL/MySQL/etc.), document storage databases (NoSQL, MongoDB, etc.), Graph databases (Neo4J, etc.)
• Experience with cloud services such as AWS S3, Athena, Glue, Lambda, SQS, or Azure Blob, Data Factory, Functions, Storage Queues, etc.
• Experience with data governance and data quality
• Experience with CI/CD, such as Gitlab CI/CD, GitHub Actions, Tekton, Jenkins, Docker, etc.
• Experience with Test Driven Development and Test Frameworks, such as pytest or unittest
• Experience with Data Catalog tools, such as Collibra, Alation, etc.
• Possession of excellent verbal and written communication skills
• Masters degree