What are the responsibilities and job description for the Data Engineer position at Brilliant Infotech Inc.?
Postion: Data Engineer Location: Edison, NJ (Onsite)
Type of Employment: W2
Contract Duration: 06 Months
Job Summary
We are looking for a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data solutions. The ideal candidate should have strong experience with data integration, ETL/ELT processes, cloud data platforms, databases, and data analytics technologies.
Key Responsibilities
Type of Employment: W2
Contract Duration: 06 Months
Job Summary
We are looking for a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data solutions. The ideal candidate should have strong experience with data integration, ETL/ELT processes, cloud data platforms, databases, and data analytics technologies.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
- Develop and optimize data integration solutions for structured and unstructured data.
- Work with cloud data platforms such as AWS, Azure, or Google Cloud Platform (GCP).
- Build and maintain data warehouses, data lakes, and lakehouse solutions.
- Develop complex SQL queries, stored procedures, and data transformation processes.
- Work with big data technologies such as Spark, Databricks, or Hadoop.
- Implement data quality, validation, governance, and security processes.
- Integrate data from APIs, databases, applications, and other enterprise data sources.
- Optimize data pipelines and queries for performance, scalability, and reliability.
- Collaborate with Data Scientists, Data Analysts, Software Engineers, and business teams to understand data requirements.
- Monitor data pipelines and troubleshoot data processing and integration issues.
- Maintain technical documentation for data architecture, pipelines, workflows, and processes.
- Strong proficiency in SQL and relational databases.
- Hands-on experience with Python, PySpark, or similar programming languages.
- Experience with ETL/ELT tools and data pipeline development.
- Experience with cloud platforms such as AWS, Azure, or GCP.
- Knowledge of data warehousing concepts and technologies.
- Experience with technologies such as Snowflake, Databricks, Azure Data Factory, AWS Glue, or Redshift.
- Experience working with Apache Spark/PySpark.
- Strong understanding of data modeling, data integration, and database concepts.
- Experience with Git and CI/CD practices.
- Good understanding of data security, governance, and data quality.
- Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
- Experience with Kafka or other real-time data streaming technologies.
- Knowledge of dbt, Airflow, or similar data orchestration tools.
- Experience with data visualization tools such as Power BI or Tableau.
- Experience working with modern cloud data architectures and lakehouse platforms.
- Relevant certifications in AWS, Azure, GCP, Databricks, or Snowflake.