What are the responsibilities and job description for the Lead Data Engineer position at ClinDCast, LLC?
< class="PDq2pG_selectionAnchorContainer" data-start="138" data-end="158">Job Summary
We are seeking an experienced Data Engineering Architect / Lead Data Engineer to design, build, and optimize scalable data platforms and pipelines. The ideal candidate will have extensive experience in ETL/ELT development, cloud-based data engineering, real-time and batch data processing, and modern data warehouse architectures. This role requires strong expertise in designing end-to-end data solutions that are scalable, secure, and aligned with enterprise standards.
Required Skills & Qualifications
- Advanced expertise in designing and developing ETL/ELT pipelines.
- Strong experience with batch data processing and near real-time/streaming data pipelines.
- Hands-on experience working with structured and semi-structured data.
- Strong knowledge of:
- Incremental data loading
- Change Data Capture (CDC)
- Pipeline orchestration and dependency management
- Strong programming skills in Python (preferred), Scala, or Java.
- Experience optimizing large-scale data processing workloads for performance and cost.
- Solid understanding of data modeling concepts:
- Star Schema
- Snowflake Schema
- Normalized and denormalized data models
- Hands-on experience with at least one major cloud platform:
- Microsoft Azure
- Amazon Web Services (AWS)
- Google Cloud Platform (Google Cloud Platform)
- Strong experience with modern data warehouses such as:
- Snowflake
- Azure Synapse
- Google BigQuery
- Amazon Redshift
Key Responsibilities
< data-start="1661" data-end="1705">Data Architecture & Solution Design- Design end-to-end data engineering architectures for enterprise-scale solutions.
- Develop scalable architectures for:
- Data Lakes and Lakehouse platforms
- Enterprise Data Warehouses
- Streaming and real-time data processing systems
- Ensure solutions align with enterprise architecture, security, governance, and compliance standards.
- Review and approve technical designs and implementation strategies.
- Lead the design and development of scalable ETL/ELT pipelines.
- Build and manage data ingestion pipelines for both batch and real-time data.
- Process structured and semi-structured data efficiently.
- Optimize data pipelines for performance, reliability, scalability, and cost.
- Manage schema evolution, metadata, and pipeline dependencies.
- Establish and enforce data quality standards and validation frameworks.
- Implement monitoring, alerting, logging, and observability for data pipelines.
- Perform root cause analysis and resolve data-related production issues.
- Drive operational excellence by improving system stability and reliability.
- Build and maintain CI/CD pipelines for data engineering workloads.
- Automate testing, deployment, and rollback processes.
- Improve platform reliability and deployment efficiency through automation and DevOps best practices.
Preferred Qualifications
- Experience with modern DataOps and CI/CD practices.
- Knowledge of data governance, security, and compliance frameworks.
- Experience designing enterprise-scale cloud-native data platforms.
- Strong analytical, troubleshooting, and communication skills.