Blog - Real-time Data Streaming, Real-time Data Processing, Real-time Data Analytics | Data Engineering Solution in Bangalore | Apache Kafka Streaming Solutions in Bangalore | Kafka Confluent Cloud Solutions in Bangalore | Kafka Streaming Implementation Support in Bangalore | Apache Kafka Support in Bangalore | Multinode Kafka Cluster Setup in Bangalore | Kafka Application Consulting in Bangalore | Kafka cloud implementation in Bangalore | Kafka infrastructure consulting in Bangalore | Kafka security implementation in Bangalore | Kafka upgrade support in Bangalore | Zookeeper setup support in Bangalore | Zookeeper Solutions in Bangalore | Multinode Zookeeper Setup in Bangalore | Big Data Consulting Service Providers in Bangalore | Data Analytics Consutling Services in Bangalore | Big Data Solution Providers in Bangalore | Big Data Analytics Companies in Bangalore | Data Analytic Services in Bangalore | Big Data Services in Bangalore | Big Data Analytics Solutions in Bangalore | Big Data Analytics Service Providers in Bangalore | Big Data Case Studies | Big Data Companies in Bangalore | Multi Node Hadoop Cluster | Data Lake creation and support | Data Ingestion Services in Bangalore | Koolanch | Artificial Intelliegence Solutions in Bangalore | Predictive Analysis Solution in Bangalore | Machine Learning Solution in Bangalore | Deep Learning Solutions Bangalore | ChatBots for Websites | Text to Speech API | DialogFlow ChatBots | ChatBots using DialogFlow | AI based image processing | AI solution providers in Bangalore | AI based Predictive Analytics | Conversational Bots Development in Bangalore | AI chatbots and voicebots | E-Commerce Solution Providers in Bangalore | Demandware Consulting Service in Bangalore | Demandware Companies in Bangalore | SFCC Consulting Service in Bangalore | SFCC Consulting Companies in Bangalore | SFCC Service Providers in Bangalore | Demandware Contract Staffing in Bangalore | Salesforce Commerce Cloud Consulting Services in Bangalore | SFCC Contract Staffing in Bangalore | Salesforce Commerce Cloud Contract Staffing in Bangalore | Oracle Consulting Services in Bangalore | Oracle Service Providers in Bangalore | Oracle Contract Staffing in Bangalore | OCC Contract Staffing in Bangalore | Oracle Commerce Cloud Consulting in Bangalore | Oracle Commerce Cloud Companies in Bangalore | SAP Hybris Consulting Services in Bangalore | SAP Hybris Service Providers in Bangalore | SAP Hybris Contract Staffing in Bangalore | SAP Hybris Commerce Cloud Consulting in Bangalore | SAP Hybris Companies in Bangalore | SAP Hybris Solutions in Bangalore | Hybris Commerce Solution in India | Hybris Solution Provider Companies | Magento Consulting Services in Bangalore | Magento Service Providers in Bangalore | Magento Contract Staffing in Bangalore | Magento Commerce Cloud Consulting in Bangalore | Magento Companies in Bangalore | Mobile App Development Company in Bangalore | Android App Development Services in Bangalore | Location Tracking Based Mobile App Development | Mobile App Development In Bangalore | Mobility Solution Provider in Bangalore | SQL Server Support Services in Bangalore | SQL Server Support Companies in Bangalore | Data Mining Solution in Bangalore | Custom App Development in Bangalore | Contract Staffing Solution in Bangalore

01JunJune 1, 2023

Knowing and valuing Apache Kafka’s ISR (In-Sync Replicas)

To get more clarity about ISR in Apache Kafka, We should first carefully examine the replication process in the Kafka broker. In short, replication means having multiple copies of our data spread across multiple brokers. Maintaining the same copies of data in different brokers makes possible the high availability in case one or more brokers go down or are untraceable in a multi-node Kafka cluster to server the requests. Because of this reason, it is mandatory to mention how...

By Gautam GoswamiApache Kafka, Data EngineeringApache Kafka, Apache Zookeeper, In-Sync Replica, Kafka Topics, multi node cluster, replicationComments Off

10AprApril 10, 2023

Handling bad messages via DLQ by configuring JDBC Kafka Sink Connector

Any trustworthy data streaming pipeline needs to be able to identify and handle faults. Exceptionally while IoT devices ingest endlessly critical data/events into permanent persistence storage like RDBMS for future analysis via multi-node Apache Kafka cluster. (Please click here to read how to setup multi-node Apache Kafka Cluster). There could be scenarios where IoT devices might send fault/bad events due to various reasons at the source points and henceforth appropriate actions can be executed to correct it further. The Apache Kafka...

By Gautam GoswamiApache Kafka, Data EngineeringApache Kafka, Big Data, JDBC Sink Connector, Kafka connect, schema registryComments Off

19FebFebruary 19, 2023

Streaming Data to RDBMS via Kafka JDBC Sink Connector without leveraging Schema Registry

In today’s M2M (Machine to machine) communications landscape, there is a huge requirement for streaming the digital data from heterogeneous IoT devices to the various RDBMS for further analysis via the dashboard, triggering different events to perform numerous actions. To support the above scenarios Apache Kafka acts like a central nervous system where data can be ingested from various IoT devices and persisted into various types of the repository, RDBMS, cloud storage, etc. Besides, various types of data pipelines...

By Gautam GoswamiApache Kafka, Data EngineeringApache Kafka, JDBC Sink Connector, Kafka connect, Kafka Topics, schema registryComments Off

09JanJanuary 9, 2023

Resolve Apache Kafka starting issue installed on Single/Multi-node cluster

This short article explains how to resolve the error “ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)kafka.common.InconsistentClusterIdException:” when we start the Apache Kafka installed and configured on a multi-node cluster. You can read here the steps for setting up multi-node Apache Kafka cluster. Without integrating Apache Zookeeper, Kafka alone won’t be able to form the complete Kafka cluster. Because ZooKeeper handles the leadership election of Kafka brokers and manages service discovery as well as cluster topology. Also tracks when topics are...

By Gautam GoswamiApache Kafka, Data Engineeringadding new broker into a kafka cluster, additional zookeeper server entry, Apache Kafka, Apache Kafka starting issue, Apache Zookeeper, configuration changes on Zookeeper, ERROR Fatal error during KafkaServer startup, Multi-node kafka cluster, Prepare to shutdown (kafka.server.KafkaServer)kafka.common.InconsistentClusterIdException, Resolving Apache Kafka starting issue, Resolving Apache Kafka starting issue on Multi Node Cluster, Resolving Apache Kafka starting issue on Single Node Cluster, single node Kafka broker, Updating Cluster ID in meta properties file, updating server.properties in kafkaComments Off

06JanJanuary 6, 2023

Few intrinsic of Apache Zookeeper and their importance

As a bird’s eye view, Apache Zookeeper has been leveraged to get coordination services for managing distributed applications. Holds responsibility for providing configuration information, naming, synchronization, and group services over large clusters in distributed systems. To consider as an example, Apache Kafka uses Zookeeper for choosing their leader node for the topic partitions. Please click here if you want read on how to setup the multi-node Apache Zookeeper cluster on Ubuntu/Linux zNodes The key concept of the Zookeeper is the znode which can be acted...

By Gautam GoswamiApache Hadoop, Apache Kafka, Architecture, Data Engineering, Hadoop Eco SystemApache Kafka, Apache Zookeeper, Big Data, Concept of zNodes, data log directory, dataDir parameter, dataLogDir parameter, distributed file systems, ephemeral zNodes, Hadoop, how to setup the multi-node Apache Zookeeper, Importance of Apache Zookeeper, persistence zNodes, receive notifications about changes to the ZooKeeper ensemble through watches, sequentialzNodes, Usage of Apache Zookeeper in Kafka Multi Node CLuster, Using Apache Zookeeper for managing distributed applications, ZooKeeper Data Directory, ZooKeeper ensemble, Zookeeper QuorumComments Off

16NovNovember 16, 2022

Overcome LEADER_NOT_AVAILABLE error on Multi-node Apache Kafka Cluster

Kafka Connect assumes a significant part in streaming data between Apache Kafka and other data systems. As a tool, it holds the responsibility of a scalable and reliable way to move the data in and out of Apache Kafka. Importing data from the Database set to Apache Kafka is surely perhaps the most well-known use instance of JDBC Connector (Source & Sink) that belongs to Kafka Connect. This short article aims to elaborate on the steps on how can we...

By Gautam GoswamiApache Kafka, Data Engineering, Multi Node Hadoop Cluster SetupApache Kafka, Importing data from the Database, JDBC Sink Connector, Kafka cluster network, Kafka connect, LEADER_NOT_AVAILABLE, message consumer, message producer, multi-broker Apache Kafka Cluster, Multi-node Apache Kafka Cluster, Multi-node kafka cluster, MySQL Database to Kafka topic, streaming dataComments Off

02OctOctober 2, 2022

Crafting a Multi-Node Multi-Broker Kafka Cluster- A Weekend Project

For the past couple of years, there has been a huge development in the appropriation of Apache Kafka. Kafka is a scalable pub/sub system and in a nutshell, is designed as a distributed multi-subscription system where data persists to disks. On top of it as a highlight, Kafka delivers messages to both real-time and batch consumers at the same time without performance degradation. Current users of Kafka incorporate Uber, Twitter, Netflix, LinkedIn, Yahoo, Cisco, Goldman Sachs, and so forth....

By Gautam GoswamiApache Kafka, Data EngineeringApache Kafka, Current users of Kafka, how to enable SSH on Ubuntu, how to produce and consume messages, install and configure the multi-node multi-broker Kafka cluster, Kafka Architecture Development Kafka Architecture Development, Kafka service, Kafka Topics, know about Apache Kafka, multi-broker kafka cluster, Multi-node kafka cluster, Multi-Node Multi-Broker Kafka Cluster, multi-node Zookeeper cluster, Number of Kafka Partic, single Kafka broker, starting the kafka server, What is Apache Kafka, Zookeeper, Zookeeper InstallationComments Off

05SepSeptember 5, 2022

Orchestrating Multi-Brokers Kafka Cluster through CLI Commands

This short article aims to highlight the list of commands to manage a running multi-broker multi-topic Kafka cluster utilizing built-in scripts. These commands will be helpful/beneficial when the cluster is not integrated or hooked up with any third party administrative tool having GUI facilities to administer or control on the fly. Of course, most of them are not free to use. Can refer here to set up a multi-broker Kafka cluster.By executing the built-in scripts available inside the bin...

By Gautam GoswamiApache Kafka, Data EngineeringApache Kafka, Big Data, bin directory of the Kafka, Kafka, Kafka Administration, list of commands to run multi-broker multi-topic Kafka cluster, modify a Kafka topic, number of replication factor can’t be higher than the number of broker in the cluster, Orchestrating Multi-Brokers Kafka Cluster, Orchestrating Multi-Brokers Kafka Cluster through CLI Commands, retention time of a Kafka topic, retrieve log directories information from the kafka brokersComments Off

25AugAugust 25, 2022

Importance of Schema Registry on Kafka Based Data Streaming Pipelines

Needless to say Apache Kafka delivers messages to both real-time and batch consumers without performance degradation and in addition to that gaining enormous momentum as a foremost component for data streaming pipelines too. Credit card fraud detection, predictive maintenance, or real-time analytics, building streaming IoT platform, etc are the example of real-time use cases. To handle massive amounts of data ingestion, Apache Kafka is the cornerstone of a robust IoT data platform. A schema defines the structure of the data...

By Gautam GoswamiApache Kafka, Architecture, Data Engineering, Data IngestionApache Kafka, assign schema info in the schema registry, Avro, building streaming IoT platform, centralized schema management, Confluent Schema Registry, Credit card fraud detection, Data ingestion, data pipeline, Data Pipelines, Data Streaming, deserialized the messages, distributed storage layer for schemas, Hadoop, JSON Schema, Kafka based data pipeline, Kafka based data streaming pipelines, Kafka connect, Kafka producers and consumers, multi-broker Kafka topic, Multi-node kafka cluster, predictive maintenance, producer-consumer contract, Protobuf schemas, real-time analytics, schema change history, schema evolution, schema of registered data streams, schema registry, Schema Registry on Kafka Based Data Streaming Pipelines, Schema Registry on Kafka Streaming Pipelines, service layer for metadata, streaming applications, streaming data to Kafka topicComments Off

18AugAugust 18, 2022

Why Kappa Architecture for processing of streaming data. Have competence to superseding Lambda Architecture?

Data is quickly becoming the new currency of the digital economy, but it is useless if it can’t be processed. The processing of data is essential for subsequent decision-making or executable actions either by the human brain or various devices/applications etc. There are two primary ways of processing data namely batch processing and stream processing. Typically batch processing has been adopted for very large data sets and projects where there is a necessity for deeper data analysis, on the...

By Gautam GoswamiArchitecture, Data Engineering, Data IngestionAmazon Kinesis, Apache Flink, Apache Hadoop, Apache Kafka, Apache Samza, Apache Storm, batch processing layer, Big Data, Data Lake, data warehouse, event based Kappa architecture, event streaming platform, Hadoop Data Lake, HDFS, Kafka, Kafka Architecture Development Kafka Architecture Development, Kappa Architecture, Kappa Architecture for streaming data, Lambda Architecture, Map-Reduce framework, messaging engine, multiple stream processors, stream processing application, streaming computation system, streaming data analyticsComments Off