What are the responsibilities and job description for the Senior Data Engineer position at Script2IT?
Job Details
Position: Senior Data Engineer
Location: Indianapolis, IN (person to travel to Indianapolis, Indiana to meet with client when required)
Duration: 3 months Contract to hire
Looking for people who does not require visa sponsorship.
Key Responsibilities:
- Experience in designing and implementation of scalable data pipelines and lakehouse architectures using Databricks
- Experience in defining best practices for data ingestion, transformation, and storage
- Experience in architecting Databricks solutions on Azure, AWS
- Lead the deployment and configuration of Databricks including Unity Catalog
- Experience in optimizing Spark jobs and cluster performance
- Implement Delta Lake for reliable and performant data lakes
- Implement role-based access controls and audit mechanisms
- Evaluate emerging technologies and tools to enhance the Databricks ecosystem
- Continuously improve performance, cost-efficiency, and scalability of data solutions
Skills and Competencies
- Lakehouse Architecture Design
- Real-Time Media Content Analytics
- MLOps Pipeline Orchestration
- Delta Lake
- Unity Catalog
- MLflow
- Databricks SQL
- Apache Kafka
- Terraform
Required Qualifications:
- Bachelor s or Master s degree in Computer Science, Engineering, or related field.
- 7 years of experience in data engineering or architecture roles.
- 3 years of hands-on experience with Databricks and Apache Spark.
- Strong proficiency in Python, SQL, and Spark.
- Experience with Delta Lake, MLflow, and Unity Catalog.
- Familiarity with CI/CD pipelines and DevOps practices in data environments.
- Certifications in Databricks, Azure/AWS/Google Cloud Platform.
- Experience with real-time data processing (Kafka, Structured Streaming).
- Knowledge of machine learning workflows and MLOps.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.