What are the responsibilities and job description for the Data Engineer (GCP - Google Cloud Platform) position at Compunnel Inc.?
Job Title :: GCP Data Engineer
Location :: Lisle, IL or Columbia, MD (4 Days in Office)
Duration :: 6 months Contract to hire or complete Full time.
Responsibilities
• Work closely with various business, IT, Analyst and Data Science groups to collect business requirements.
• Design, develop, deploy and support high performance data pipelines both inbound and outbound.
• Model data platform by applying the business logic and building objects in the semantic layer of the data platform.
• Optimize data pipelines for performance, scalability, and reliability.
• Implement CI/CD pipelines to ensure continuous deployment and delivery of our data products.
• Ensure quality of critical data elements, prepare data quality remediation plans and collaborate with business and system owners to fix the quality issues at its root.
• Document the design and support strategy of the data pipelines
• Capture, store and socialize data lineage and operational metadata
• Troubleshoot and resolve data engineering issues as they arise.
• Develop REST APIs to expose data to other teams within the company.
• Mentor and guide junior data engineers.
Education
• Required - Bachelor's degree in Computer Science, Computer Engineering, Software Engineering, or other related technical field
• Nice to Have - Master’s Degree in Computer Science, Computer Engineering, Software Engineering, or other related technical field
Work Experience
• Minimum 6 years of experience in data engineering solutions such as data platforms, ingestion, data management, or publication/analytics
• 2 years of experience in Google cloud with services like BigQuery, Composer, GCS, DataStream, Dataflows
Knowledge, Skills and Abilities
• Expert knowledge on SQL and Python programming
• Experience working with Airflow as workflow management tools and build operators to connect, extract and ingest data as needed.
• Experience in tuning queries for performance and scalability
• Experience in Real Time Data ingestion using GCP Pub Sub, Kafka, Spark or similar.
• Excellent organizational, prioritization and analytical abilities
• Have proven experience working in incremental execution through successful launches.
• Excellent problem-solving and critical-thinking skills to recognize and comprehend complex data issues affecting the business environment.
• Experience working in agile environment.