Data Integration Engineer | Secret Cleared (JOB ID 002983)

Innovien Solutions is looking for a driven Senior Data Integration Engineer to join a team spearheading mission-critical federal network operations that securely connect millions of end users across thousands of DoD and federal sites worldwide. In this role, you will help lead data integration development, expanding a foundational Integrated Data Architecture platform and delivering modern analytics capabilities in support of large-scale network operations.

REQUIREMENTS:

• 5 years of hands-on data integration experience – developing & implementing Kafka integrations between ELK/Elastic Stack (Elasticsearch, Logstash, Kibana)
• Including integrating RESTful APIs, connectors, and event streaming pipelines
• 3 years of Databricks experience building pipelines with Delta tables, data cleansing, and implementing Bronze/Silver/Gold (medallion) architecture
• Extensive Python and/or Java programming experience
• Experience working in Agile development environment (SDLC) supporting sprint cycles, testing, and deployment activities
• Active SECRET Clearance or higher REQUIRED
• CompTIA Security certification required within the first 14 days of employment

PREFERRED SKILLSET:

• Certified Confluent Developer and/or Certified Elastic Engineer Certifications
• Kubernetes containerization experience for cloud deployments and/or AWS Gov Cloud environment experience
• Experience developing and deploying software in a DoD environment
• Experience with Atlassian tools including JIRA and Confluence

RESPONSIBILITIES:

• Develop and implement integration solutions using Kafka and Elastic as the primary data architecture platforms, with expanded integration to other technologies, including but not limited to Databricks.
• Integrate data sources into Databricks, Confluent (Kafka), and Elastic platforms in support of a core data and analytics environment
• Develop Kafka system integrations, custom connectors, and work with ksqlDB and Kafka Streams for data processing based on the design solution
• Design and implement solutions within Databricks, including Delta Lake tables and medallion architecture layers, to support analytics and data transformation initiatives
• Develop Kafka-based integrations into Databricks and Elastic, including custom connectors, APIs, and event streaming pipelines
• Support the integration of new data sources into the existing platform environment
• Support and maintain Elasticsearch and Logstash integrations to ensure continuity and stability of the existing ELK environment
• Automate the full software lifecycle from design and development through testing and deployment, including production environments
• Work with other members of the data integration team to propose solutions based on mission needs and platform strategies
• Develop DoD requirements, traceability, and detailed plans and schedules, including software systems engineering and interface documents (IDDs/ICDs)
• Interact with the customer to address data engineering technical considerations and associated problems, issues, or conflicts