Data Science

20Nov

The Zero Copy Principle With Apache Kafka

The Apache Kafka, a distributed event streaming technology, can process trillions of events each day and eventually demonstrate its tremendous throughput and low latency. That’s building trust and over 80% of Fortune 100 businesses use and rely on Kafka. To develop high-performance data pipelines, streaming analytics, data integration, etc., thousands of companies presently use Kafka around the globe. By leveraging the zero-copy principle, Kafka improves efficiency in terms of data transfer. In short, when doing computer processes, the zero-copy...

By Apache Kafka, Data Science, , 0 Comments
Read more...
Causes and remedies of poison pill in Apache Kafka
25Sep

Causes and remedies of poison pill in Apache Kafka

A poison pill is a message deliberately sent to a Kafka topic, designed to consistently fail when consumed, regardless of the number of consumption attempts. Poison Pill scenarios are frequently underestimated and can arise if not properly accounted for. Neglecting to address them can result in severe disruptions to the seamless operation of an event-driven system. The poison pill for various reasons: The failure of deserialization of the consumed bytes from the Kafka topic on the consumer side. Incompatible serializer and deserializer...

By Apache Kafka, Data Engineering, Data Science, , , , , , , , , Comments Off
Read more...
06Jul

The significance of deep storage in Apache Druid

The phrase “deep storage” refers to the long-term storage system used by Apache Druid, where past data segments are preserved for durability and retrieval in the future. Druid stores data in files called segments and deep storage is the place where segments are stored. Even though Druid’s native integration with Apache Kafka (can read here how to integrate Druid with Kafka) and Amazon Kinesis, which allows query-on-arrival at millions of events per second, low latency ingestion, etc., and eventually enables us to...

By Data Engineering, Data Science, , Comments Off
Read more...
12Aug

The Brain and Convolutional Neural Network (CNN)

In the beginning of our life parents/family used to tell us name of the objects we see. We learned by examples given to us. Slowly we started recognizing certain things more often. They became so common that next time we saw them we would instantly know the name of the object and details of that. We label every object based on what we have learnt in the past. This is what we subconsciously do all days, we see, label, make...

By Artificial Intelligence, Data Science, Deep Learning, Machine Learning, , , , , , , , , , , , , , , , , , , , , , , , , , , , , Comments Off
Read more...