Confluent Kafka to Apache Kafka Migration
For one of the major IOT/IIOT and M2M manufacturing companies, that develops and manufactures end-to-end Remote Monitoring/Energy Management solutions
This solution is for the existing platform of the customer which had confluent Kafka, needed to be migrated & integrated with Apache Kafka, in order to collect device data from controller, sensors, meters installed on telecom tower sites and eventually pushes to cloud / on premises infrastructure.
challenge
Installation and integration of multi-node Kafka cluster as a middleware data streaming platform to consume the real time device data from various telecom towers and dump to RDBMS, so that monitoring team can take the appropriate action by visualizing the web based dash board
solution
- Design and architecture of the proposed Kafka cluster with optimizations
- Implementation and configuration of scalable Kafka Cluster
- Installation & configuration of Kafka Connect to ingest device data from Kafka cluster to MySQL Database
- Accumulating and ingesting real time monitoring data from IOT devices by leveraging Apache Kafka framework and subsequent processing and rendering into the dashboard.
- Tuning & Scale-up
- Testing & Support
outcome
- Target Kafka Configuration
- Cluster Sizing
- Desired Throughput