What are the responsibilities and job description for the Senior Data Engineer - SME position at Janus Soft Inc?
Senior Data Engineer
Job Role
The Senior Data Engineer develops new tools, code, and services to execute data engineering activities involving data of varying types and in varying conditions. The role involves movement of structured and unstructured data, executing data ingestion activities, developing code to format data, analyzing source data formats, and developing ETL code.
Key responsibilities include:
- Designing and optimizing Data Pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi and Kubernetes containers
- Ensuring the pedigree and provenance of the data is maintained
- Cleaning and preprocessing data to enable access for advanced analytics
- Collaborating with enterprise working groups to advance the state of data standards
- Collaborating with the engineering team, data stewards, and mission partners
- Collaborating with software engineers to update, configure, and maintain data services
- Ensuring data quality by working with the testing and data quality team
- Adapting to various types and formats of data, and working with development teams to integrate new data processing platforms
Mandatory Skills
- 10 years' experience with:
- Data lifecycle engineering
- Development and maintenance of extract, transform and load (ETL) tools and services
- Cloud and on-prem data storage and processing solutions
- Python, SQL, Spark and other data engineering programming
- COTS and open source data engineering tools such as ElasticSearch and NiFi
- Processing data within the Agile Lifecycle
