What are the responsibilities and job description for the Data Engineer position at Intellect Quest LLC?
Position: Data Engineer
Only and USC
It is a 6-month contract-to-hire, or 3-year long remote.
It is 4 days on-site/1 day remote.
Top Skills - Must Haves
- Development
- SQL
- Power bi
- Dashboarding
- Business intelligence
- Analysis
- Data
- Data model
- Dashboard
- Data analysis
- Reporting
- Azure
- Python
- Databricks
- Medallion architecture
Top Skills'' Details
1. Experience creating a pipeline
2. ETL-strong
3. Helping curate data
1. Create data models and Pipeline
2. Strong Azure, SQL, Python, and Databricks experience
3. Ability to analyze data, create dashboards, charts and make data more usable
Job Description
The Power Delivery Organization of Southern Company is launching a new team as part of their "Leading as a System" initiative, and this specific area will be focused on Generation. Working very closely with Databricks to get things up and running for their team. Databricks should have an instance up and running for them and in UA testing by mid mid February, so they need a data engineer to help begin to ramp up their team. As part of that initial ramp up, they would like to hire this Data Engineer as a contract to hire. Because part of what "leading as a system" does is look at data from several different sources - all of the Operation cos and TO Generation, there is data coming in from all over the place, so this contractor will need to be able to analyze exactly how bad the data is, and determine how they can possibly get it back to a usable format. They will have to work with people from the TO (Technology Organization) to cultivate a pipeline in Data Bricks and develop code and script the data so that the team is able to use it. . Candidates will also need MS SQL Server Management Studio to do mapping of on prem databases because they are unsure what data for other organizations they have worked on or where to find it and so they will need to be able to pull better views and tables if possible.
Engineer should be familiar with Medallion architecture as a data design pattern (so progressing from bronze to silver to gold layers) and if they have experience in governance- especially previous Southern Company governance that would be huge (but obviously they know that would be tough to find).
5-10 Years Focused on manipulating data in a software engineering capacity. Some of that data might live in relational systems, but it is increasingly moving towards NoSQL systems and data lakes. Normalize databases and ascertain the structure of the data meets the requirements of the applications that are accessing the information. Construct datasets that are easy to analyze and support company requirements. Combine raw information from different sources to create consistent and machine-readable formats. This IT role requires a significant set of technical skills, including a deep knowledge of SQL, data modeling, and tools like Spark/Hive/Airflow.
Must have skills and experience required for creating Databricks pipelines for multiple types of sources.
Working experience with batch and real-time data processing frameworks.
Working experience with data modelling, data access, schemas, and data storage techniques.
Working experience with data quality tools.
Experience in creating functional and technical designs for data engineering and analytics solutions.
Experience implementing data models of different schemas and working with diverse data source types.
Hands-on experience developing solutions with big data technologies such as Hadoop, HIVE and Spark.
Hands-on experience developing and supporting statistical models, R, and/or Python based AI/ML solutions.
5 years hands on experience designing, developing, testing, deploying, and supporting data engineering and analytics solutions using on premises tools such as MSBI (SSIS/SSAS),
3 years hands on experience designing, developing, testing, deploying, and supporting data engineering and analytics solutions using Microsoft cloud-based tools such as Azure Data Lake, Azure Data Factory, Azure Databricks, Python, Azure Synapse, Azure Key Vault, and Power BI.
Experience with Containerization methodologies – Docker, OpenShift, etc.
Experience with Agile as well as DevOps, CI/CD methodologies.
Hands-on experience designing and developing solutions involving data sourcing, enrichment and delivery using APIs & Web Services.
Salary : $80 - $85