Must have skill set:Spark, S3, Glue, AWS Redshift , python and stream set exp

6-8 years of IT experience focusing on enterprise data architecture and management.

Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling

Experience with Databricks & on Prem , Structured Streaming, Delta Lake concepts, and Delta Live Tables required

Experience with Spark scala

Data Lake concepts such as time travel and schema evolution and optimization

Structured Streaming and Delta Live Tables with Databricks a bonus

Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support

Advanced level understanding of streaming data pipelines and how they differ from batch systems

Formalize concepts of how to handle late data, defining windows, and data freshness

Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc