What are the responsibilities and job description for the Sr. Data Engineer position at CX Data Labs Pvt Ltd?
Job Details
Data Engineer with DataBricks Experience
Dallas, TX
Fulltime
Minimum Years of Experience: 8 Years
Key Responsibilities:
- Develop, maintain, and optimize scalable ETL/ELT pipelines using PySpark and Databricks.
- Collaborate with cross-functional teams to design and implement data models and data integration solutions.
- Create and maintain robust SQL scripts for querying, transforming, and analyzing data.
- Work on Databricks to manage big data workloads and ensure optimal performance for large-scale datasets.
- Ensure data quality, integrity, and governance across the organization s data assets.
- Automate data workflows and deploy reliable solutions for real-time data processing.
- Debug and troubleshoot performance issues with data pipelines and implement enhancements.
- Stay up-to-date with emerging trends and best practices in data engineering and big data technologies.
Required Skills and Qualifications:
Technical Skills:
- Proficiency in SQL for complex queries, database design, and optimization.
- Strong experience with PySpark for data transformation and processing.
- Hands-on experience with Databricks for building and managing big data solutions.
- Familiarity with cloud platforms like AWS, Azure, or Google Cloud.
- Knowledge of data warehousing concepts and tools (e.g., Snowflake, Redshift).
- Experience with data versioning and orchestration tools like Git, Airflow, or Dagster.
- Solid understanding of Big Data ecosystems (Hadoop, Hive, etc.).
Preferred Qualifications:
- 8 years of relevant work experience in data engineering or software engineering equivalent.
- 5 years of experience in implementing big data processing technology: AWS / Azure / Google Cloud Platform, Apache Spark, Python.
- Experience writing and optimizing SQL queries in a business environment with large-scale, complex datasets.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
