What are the responsibilities and job description for the Data Engineer position at Aerstone?
Position Overview:
We are seeking a skilled and motivated Data Engineer / Database Engineer to support our growing team providing an enterprise data warehouse and visualization environment. The ideal candidate will have hands-on experience with Microsoft SQL Server, ETL pipelines, and cloud-based data ingestion tools—especially AWS Glue. You will be responsible for maintaining and enhancing our database systems, enabling near real-time data pipelines from source databases (Oracle, SQL, etc) to Microsoft SQL Server, and ensuring secure, efficient, and scalable data access.
Key Responsibilities:
- Support and maintain Microsoft SQL Server environments, with flexibility to adapt to alternative database platforms as the architecture evolves.
- Design and implement data pipelines using AWS Glue for efficient and scalable data ingestion.
- Build and maintain a near real-time data ingestion framework from Oracle databases to Microsoft SQL Server.
- Develop and manage ETL processes, including incremental/delta-based extraction of raw data from source systems.
- Build and optimize relational database schemas, ensuring performance, scalability, and data integrity.
- Implement and manage fine-grained roles and permissions in SQL Server to enforce data security and access control policies.
- Work closely with Data Analysts, Software Engineers, and other stakeholders to ensure data systems meet business requirements.
- Monitor and troubleshoot data pipelines and workflows to ensure reliability and consistency.
Qualifications:
- Proven experience with Microsoft SQL Server administration and development.
- Proficiency in ETL/ELT concepts and experience with tools like AWS Glue, Apache Spark, or equivalent.
- Solid understanding of Oracle database systems and data extraction methodologies (especially delta/change data capture techniques).
- Experience with cloud data platforms, particularly AWS (S3, Glue, RDS, etc.).
- Strong SQL skills and experience with role-based access control in database systems.
- Familiarity with scripting or programming languages (e.g., Python, Shell) for automation and data processing.
- Ability to work independently and communicate effectively in a collaborative team environment.
Preferred Skills (Not Required):
- Proficiency with Oracle GoldenGate or equivalent CDC (Change Data Capture) tools for real-time replication.
- Experience with modern data architectures (e.g., Data Vault 2.0).
- Knowledge of CI/CD practices for data engineering.
- Exposure to data cataloging and metadata management tools.