Kafka Expert – Data Streaming Engineer
We are seeking a highly skilled Kafka Expert with deep expertise in Apache Kafka, Linux systems, and Kubernetes to join our data platform team. This is a hands-on engineering role focused on building and optimizing real-time data streaming solutions that are scalable, secure, and production-ready. You’ll work closely with infrastructure and development teams to design streaming architectures, enable robust use cases, and ensure high availability and performance across distributed systems.
Key Responsibilities
-
Design, deploy, and manage Apache Kafka clusters in development, testing, and production.
-
Optimize Kafka performance, reliability, and scalability for high-throughput pipelines.
-
Integrate Kafka with tools such as Kafka Connect, Schema Registry, MirrorMaker, and Kafka Streams.
-
Deploy and operate Kafka on Kubernetes using Helm, Operators, or custom manifests.
-
Manage and troubleshoot Linux systems (Red Hat, Debian, Ubuntu) supporting Kafka.
-
Build automation and Infrastructure-as-Code (IaC) using Ansible, Terraform, GitLab CI/CD, and GitOps practices.
-
Implement observability and alerting with Prometheus, Grafana, Fluent Bit, Spark UI, and Flink Dashboard.
-
Lead incident response and root cause analysis for Kafka and related systems.
-
Collaborate with teams to build end-to-end data pipelines for real-time integration, event-driven microservices, and logging/monitoring.
-
Design and operate Spark Structured Streaming and Flink DataStream pipelines, including:
-
Resource allocation, scaling, and job scheduling.
-
State management, checkpointing, and fault tolerance.
-
Performance tuning for low latency and high throughput.
Required Qualifications
-
5+ years administering and supporting Apache Kafka in production.
-
Strong Linux system administration skills (Red Hat, Debian, Ubuntu).
-
Solid experience with Kubernetes (OpenShift, Rancher, or upstream distributions).
-
Proficiency in scripting and automation (Bash, Python, Ansible).
-
Experience with IaC, CI/CD pipelines, and GitOps.
-
Strong background in Kafka security, monitoring, and schema management.
-
4+ years building scalable pipelines with Apache Spark and/or Apache Flink, with expertise in performance tuning and distributed systems.