What are the responsibilities and job description for the Big Data Engineer position at Inclusion Cloud?
About the Role
We are seeking a Senior Big Data Engineer with strong experience in Business Intelligence, Machine Learning, and AI-driven solutions to join our data and analytics team. This role is ideal for a hands-on professional who can design, build, and optimize large-scale data platforms while enabling advanced analytics and AI use cases.
The candidate must be able to communicate fluently in English, as they will work closely with international stakeholders, data scientists, product teams, and business users.
Key Responsibilities
Design, build, and maintain scalable Big Data architectures for analytics, BI, and AI use cases.
Develop and optimize batch and streaming data pipelines, including real-time ingestion using Kafka, to process large volumes of data from multiple sources.
Design and manage Kafka-based event-driven and streaming architectures.
Enable Business Intelligence solutions, ensuring data availability, quality, and performance for dashboards and reports.
Collaborate with Data Scientists and ML Engineers to prepare data for Machine Learning and AI models.
Support the deployment, monitoring, and optimization of ML and AI models in production environments.
Work closely with business and analytics teams to translate business requirements into technical data solutions.
Ensure data governance, security, and compliance across data platforms.
Optimize data performance, cost, and reliability in cloud-based environments.
Document data models, pipelines, and architectural decisions.
Required Skills & Experience
5 years of experience in Big Data Engineering or similar data-focused roles.
Strong hands-on experience with Apache Kafka, including:
Designing and implementing streaming pipelines
Working with topics, producers, consumers, and schemas
Supporting real-time or near real-time data processing
Strong experience with Big Data technologies such as Spark, Hadoop, Databricks, and related ecosystems.
Solid background in Business Intelligence, including data modeling and support for reporting and dashboards.
Hands-on experience working with Machine Learning and AI use cases, including data preparation and model integration.
Proficiency in SQL and at least one programming language such as Python or Scala.
Experience with cloud platforms (AWS, Azure, or GCP).
Strong knowledge of data warehousing and data lake architectures.
Experience working with structured and unstructured data.
Strong problem-solving and analytical skills.
Fluent English (spoken and written) is mandatory.
Nice to Have
Experience supporting AI/ML pipelines in production.
Knowledge of MLOps concepts and tools.
Experience with real-time or streaming analytics.
Familiarity with data governance and data quality frameworks.
Experience working on Oracle-based projects (Oracle Database, Oracle Analytics, Oracle Cloud Infrastructure) is a strong plus.
Experience working in international or distributed teams.