What are the responsibilities and job description for the Senior Data Engineer with Security Clearance position at Halvik?
Requirements
The Contractor shall analyze requirements and design, code, test, debug, and document data pipelines. The Contractor shall have the ability to create and integrate application programming interfaces (APIs). Their technical skills generally include multiple programming languages and a deep knowledge of SQL database design. The data engineer role requires a more in-depth knowledge in programming for integrating complex models and using advanced software library frameworks to distribute large, clustered data sets. Data engineers collect and arrange data in a form that is useful for analytics. Data Engineers should be familiar with both streaming and batch models of data movement, aware of the trade-offs between the two, and able to apply that knowledge to solve for business and operational problems. The Contractor shall perform the following tasks:
The Contractor shall analyze requirements and design, code, test, debug, and document data pipelines. The Contractor shall have the ability to create and integrate application programming interfaces (APIs). Their technical skills generally include multiple programming languages and a deep knowledge of SQL database design. The data engineer role requires a more in-depth knowledge in programming for integrating complex models and using advanced software library frameworks to distribute large, clustered data sets. Data engineers collect and arrange data in a form that is useful for analytics. Data Engineers should be familiar with both streaming and batch models of data movement, aware of the trade-offs between the two, and able to apply that knowledge to solve for business and operational problems. The Contractor shall perform the following tasks:
- Developing, maintaining, and testing infrastructures for data generation to transform data from various structured and unstructured data sources.
- Develop complex queries to ensure accessibility while optimizing the performance of NoSQL and/or big data infrastructure.
- Create, document, and maintain optimal data pipeline architecture.
- Create, document, and maintain APIs; integrate with existing APIs where applicable.
- Create, document, and maintain data dictionaries, database and schema metadata for all data efforts.
- Build and maintain the infrastructure to support data movement -- Extraction, Transformation, and Loading (ETL), Extraction, Loading, and Transformation (ELT), Extraction, Loading, Transformation, and Loading (ELTL), as well as other data movement paradigms – of data from a wide variety of data sources.
- Extract data from multiple data sources, relational SQL and NoSQL databases, and other platform APIs, for data ingestion and integration.
- Configure and manage data analytic frameworks and pipelines using databases and tools such as (but not limited to) NoSQL, SQL, NiFi, Kafka, HDInsight, MongoDB, Cassandra, Neo4j, GraphDB, OrientDB, Spark, Flink, Hadoop, Kafka, Hive, and others.
- Apply distributed systems concepts and principles such as consistency and availability, liveness and safety, durability, reliability, fault-tolerance, consensus algorithms.
- Administrate cloud computing and CI/CD pipelines to include Azure, Google, and Amazon Web Service (AWS).
- Coordinate with stakeholders, including product, data and design teams to assist with data-related technical issues and support their data infrastructure needs Must have a current Security certification (or willing to obtain within 6 months of start date)