What are the responsibilities and job description for the Need Data Engineer - Columbus, OH position at Radiantze?
Job Title: Data Engineer
Location: Columbus, OH
Experience: 8
Location: Columbus, OH
Experience: 8
Job Summary
We are seeking a Data Engineer with strong expertise in Python, PySpark, Databricks, and AWS to design, develop, and optimize scalable data pipelines and cloud-based data platforms. The ideal candidate should have hands-on experience with Delta Lake, Unity Catalog, Delta Live Tables (DLT), Medallion Architecture, and Databricks performance optimization techniques.
Required Skills
- Strong experience with Python and PySpark for large-scale data processing.
- Hands-on experience with Databricks and AWS services.
- Expertise in building ETL/ELT pipelines using Databricks.
- Experience with Delta Lake and data lakehouse architecture.
- Hands-on experience implementing Unity Catalog for data governance, schema management, and row/column-level security.
- Experience developing data pipelines using Delta Live Tables (DLT).
- Strong understanding of Medallion Architecture (Bronze, Silver, Gold layers).
- Experience with Auto Loader for scalable file ingestion into Delta Lake.
- Knowledge of Databricks optimization techniques including:
- OPTIMIZE
- Z-ORDER
- VACUUM
- Liquid Clustering
- Experience with AWS services such as:
- S3
- Glue
- Lambda
- EMR
- Redshift
- IAM
- Strong SQL and data modeling skills.
- Experience with data quality, monitoring, and performance tuning.
Responsibilities
- Design and develop scalable data pipelines using Python, PySpark, Databricks, and AWS.
- Build and maintain Lakehouse solutions using Delta Lake.
- Implement and manage Unity Catalog for governance and security.
- Develop declarative pipelines using Delta Live Tables (DLT).
- Design Bronze, Silver, and Gold data layers following Medallion Architecture.
- Implement streaming and batch ingestion solutions using Auto Loader.
- Optimize Databricks workloads using OPTIMIZE, Z-ORDER, VACUUM, and Liquid Clustering.
- Collaborate with Data Scientists, Analysts, and Business Teams to deliver reliable data solutions.
- Ensure data security, quality, scalability, and performance across the platform.
Preferred Qualifications
- Databricks Certification.
- AWS Certification.
- Experience with CI/CD, Git, Airflow, or Terraform.
- Experience with real-time data processing and streaming frameworks.
Primary Skills: Python, PySpark, Databricks, AWS, Delta Lake, Unity Catalog, Delta Live Tables (DLT)