Yearly Archives - 2022
16Nov

Overcome LEADER_NOT_AVAILABLE error on Multi-node Apache Kafka Cluster

Kafka Connect assumes a significant part in streaming data between Apache Kafka and other data systems. As a tool, it holds the responsibility of a scalable and reliable way to move the data in and out of Apache Kafka. Importing data from the Database set to Apache Kafka is surely perhaps the most well-known use instance of JDBC Connector (Source & Sink) that belongs to Kafka Connect. This short article aims to elaborate on the steps on how can we...

By Apache Kafka, Data Engineering, Multi Node Hadoop Cluster Setup, , , , , , , , , , , , Comments Off
Read more...
02Oct

Crafting a Multi-Node Multi-Broker Kafka Cluster- A Weekend Project

For the past couple of years, there has been a huge development in the appropriation of Apache Kafka. Kafka is a scalable pub/sub system and in a nutshell, is designed as a distributed multi-subscription system where data persists to disks. On top of it as a highlight, Kafka delivers messages to both real-time and batch consumers at the same time without performance degradation. Current users of Kafka incorporate Uber, Twitter, Netflix, LinkedIn, Yahoo, Cisco, Goldman Sachs, and so forth....

By Apache Kafka, Data Engineering, , , , , , , , , , , , , , , , , , Comments Off
Read more...
05Sep

Orchestrating Multi-Brokers Kafka Cluster through CLI Commands

This short article aims to highlight the list of commands to manage a running multi-broker multi-topic Kafka cluster utilizing built-in scripts. These commands will be helpful/beneficial when the cluster is not integrated or hooked up with any third party administrative tool having GUI facilities to administer or control on the fly. Of course, most of them are not free to use. Can refer here to set up a multi-broker Kafka cluster.By executing the built-in scripts available inside the bin...

By Apache Kafka, Data Engineering, , , , , , , , , , , Comments Off
Read more...
25Aug

Importance of Schema Registry on Kafka Based Data Streaming Pipelines

Needless to say Apache Kafka delivers messages to both real-time and batch consumers without performance degradation and in addition to that gaining enormous momentum as a foremost component for data streaming pipelines too. Credit card fraud detection, predictive maintenance, or real-time analytics, building streaming IoT platform, etc are the example of real-time use cases. To handle massive amounts of data ingestion, Apache Kafka is the cornerstone of a robust IoT data platform. A schema defines the structure of the data...

By Apache Kafka, Architecture, Data Engineering, Data Ingestion, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Comments Off
Read more...
18Aug

Why Kappa Architecture for processing of streaming data. Have competence to superseding Lambda Architecture?

Data is quickly becoming the new currency of the digital economy, but it is useless if it can’t be processed. The processing of data is essential for subsequent decision-making or executable actions either by the human brain or various devices/applications etc. There are two primary ways of processing data namely batch processing and stream processing. Typically batch processing has been adopted for very large data sets and projects where there is a necessity for deeper data analysis, on the...

By Architecture, Data Engineering, Data Ingestion, , , , , , , , , , , , , , , , , , , , , , , , Comments Off
Read more...