Senior Data Engineer

LatentView Analytics is a leading global analytics and decision sciences provider, delivering solutions that help companies drive digital transformation and use data to gain a competitive advantage. With analytics solutions that provide a 360-degree view of the digital consumer, fuel machine learning capabilities and support artificial intelligence initiatives., LatentView Analytics enables leading global brands to predict new revenue streams, anticipate product trends and popularity, improve customer retention rates, optimize investment decisions, and turn unstructured data into valuable business assets

Required Qualifications:

7 years of experience in Data Engineering roles

Extensive hands-on experience with Databricks (Delta Lake, Unity Catalog, cluster management, notebooks, job orchestration)

Strong experience with AWS cloud services (S3, IAM, Glue, EMR, Lambda, Redshift, CloudWatch)

Proven expertise in Apache Airflow for workflow orchestration and scheduling

Strong programming skills in Python and SQL

Solid understanding of Apache Spark (PySpark, Spark SQL, performance tuning)

Experience with data modeling, warehousing concepts, and Lakehouse architecture

Familiarity with version control (Git) and CI/CD practices

Strong understanding of data governance, quality, and lineage principles

Excellent problem-solving and communication skills

Responsibilities:

Design, develop, and maintain scalable ETL/ELT pipelines using Databricks (PySpark/Spark SQL)

Build, orchestrate, and monitor workflows using Apache Airflow (DAG design, scheduling, dependency management)

Architect and manage data infrastructure on AWS (S3, Glue, Lambda, EMR, Redshift, IAM, etc.)

Optimize data pipelines for performance, reliability, and cost-efficiency

Implement data quality checks, testing frameworks, and observability/monitoring for pipelines

Collaborate with Data Scientists, Analysts, and Product teams to understand data requirements

Design and maintain data models (Lakehouse/Medallion architecture — Bronze/Silver/Gold layers)

Implement CI/CD pipelines for data engineering workflows

Ensure data security, access controls, and compliance across the data platform

Document architecture, pipelines, and processes for team knowledge sharing

Mentor junior data engineers and contribute to engineering best practices

Required skills:

Databricks, AWS, Python, SQL, Pyspark, Genie, GenAI, Analytics, Cybersecurity Knowledge

At LatentView Analytics, we value a diverse, inclusive workforce and provide equal employment opportunities for all applicants and employees. All qualified applicants for employment will be considered without regard to an individual's race, color, sex, gender identity, gender expression, religion, age, national origin or ancestry, citizenship, physical or mental disability, medical condition, family care status, marital status, domestic partner status, sexual orientation, genetic information, military or veteran status, or any other basis protected by federal, state or local laws.