What are the responsibilities and job description for the Data Engineer position at TalentBridgeX?
Location: Tampa, FL
Security Clearance: TS/SCI clearance required
Responsibilities:
- Design and implement a high-throughput data pipeline architecture integrating structured and unstructured logistics data into a centralized data lake
- Leverage Apache Kafka for real-time streaming data and ETL frameworks such as Apache NiFi or Apache Airflow for batch processing
- Implement data quality frameworks to ensure data integrity across systems, including data lineage tracking and audit trails
- Utilize cloud platforms (AWS S3, Azure Blob Storage, Google Cloud Storage) for scalable storage and compute
- Ensure data security with encryption at rest and in transit, utilizing AWS KMS or Azure Key Vault
Qualifications
Education:
- Bachelor's degree required
Experience:
- 5 years experience
Technical Skills:
- Proficiency in AWS or Azure
- Proficiency in Python
- Proficiency in SQL/NoSQL
- Proficiency in Spark
- Proficiency in Kafka
Security Clearance:
- TS/SCI clearance required