Kafka Administrator

Senior Kafka Administrator

St. Louis, Missouri or Denver, Colorado

The key skills focused on are:

1. Strong Terraform experience
2. CI/CD expertise (any major toolset/platform is fine)
3. Kafka experience any flavor, preferably with Confluent and Amazon MSK

Key Responsibilities:

Kafka Administration

• Deploy, configure, and manage Apache Kafka clusters and AWS MSK environments.
• Monitor broker health, partitions, replication factors, and consumer lag.
• Perform capacity planning and cluster scaling activities.
• Manage Kafka security using SSL, SASL, ACLs, and encryption standards.
• Troubleshoot producer, consumer, and broker performance issues.
• Support Kafka Connect, Schema Registry, Cruise Control, and MirrorMaker implementations.

AWS Cloud Administration

• Manage cloud infrastructure services including EC2, S3, IAM, VPC, EBS, CloudWatch, CloudTrail and AWS Glue.
• Support AWS Managed Streaming for Kafka (MSK), EMR, Lambda, and Airflow environments.
• Implement cloud security best practices and governance controls.
• Perform infrastructure provisioning and automation using Infrastructure as Code (IaC).
• Monitor cloud resource utilization and optimize operational costs.

DevOps & Automation

• Design and maintain CI/CD pipelines using Jenkins, GitHub Actions, GitLab CI, or similar tools.
• Automate infrastructure deployment using Terraform, CloudFormation, and Ansible.
• Manage source control repositories and release processes.
• Implement monitoring and alerting solutions using Prometheus, Grafana, Splunk, ELK, or CloudWatch.
• Support containerization technologies such as Docker and Kubernetes.
• Develop automation scripts using Python, Shell, or Bash.

Operations & Support

• Provide Level 2 and Level 3 production support.
• Participate in on-call support rotations and incident management activities.
• Perform root cause analysis (RCA) and implement preventive measures.
• Create and maintain operational documentation and standard operating procedures.
• Ensure compliance with security, audit, and regulatory requirements.

Required Skills

• Bachelor’s degree in Computer Science, Information Technology, or related field.
• Experience with Cloudera CDP, AWS MSK, Airflow, and Spark.
• AWS, Google Cloud Platform, Kafka, or Kubernetes certifications.
• Experience supporting large-scale production environments handling petabyte- scale data workloads.

• Apache Kafka Administration
• AWS Cloud Services
• Linux (RHEL/Rocky Linux)
• Shell Scripting and Python
• Jenkins, Git, Ansible, Terraform
• Docker and Kubernetes
• Monitoring Tools (Grafana, Prometheus, Splunk)
• Networking, Security, and High Availability Concepts
• Performance Tuning and Capacity Planning

Key Achievements Expected

• Maintain platform availability above 99.9%.
• Adopt AI-assisted engineering practices to improve operational efficiency, reduce manual effort, and accelerate troubleshooting and documentation.
• Automate repetitive operational tasks.
• Improve cluster performance and resource utilization.
• Ensure secure, scalable, and reliable data platform operations.
• Support enterprise data engineering, analytics, and AI/ML workloads efficiently.