Blog - Real-time Data Streaming, Real-time Data Processing, Real-time Data Analytics | Data Engineering Solution in Bangalore | Apache Kafka Streaming Solutions in Bangalore | Kafka Confluent Cloud Solutions in Bangalore | Kafka Streaming Implementation Support in Bangalore | Apache Kafka Support in Bangalore | Multinode Kafka Cluster Setup in Bangalore | Kafka Application Consulting in Bangalore | Kafka cloud implementation in Bangalore | Kafka infrastructure consulting in Bangalore | Kafka security implementation in Bangalore | Kafka upgrade support in Bangalore | Zookeeper setup support in Bangalore | Zookeeper Solutions in Bangalore | Multinode Zookeeper Setup in Bangalore | Big Data Consulting Service Providers in Bangalore | Data Analytics Consutling Services in Bangalore | Big Data Solution Providers in Bangalore | Big Data Analytics Companies in Bangalore | Data Analytic Services in Bangalore | Big Data Services in Bangalore | Big Data Analytics Solutions in Bangalore | Big Data Analytics Service Providers in Bangalore | Big Data Case Studies | Big Data Companies in Bangalore | Multi Node Hadoop Cluster | Data Lake creation and support | Data Ingestion Services in Bangalore | Koolanch | Artificial Intelliegence Solutions in Bangalore | Predictive Analysis Solution in Bangalore | Machine Learning Solution in Bangalore | Deep Learning Solutions Bangalore | ChatBots for Websites | Text to Speech API | DialogFlow ChatBots | ChatBots using DialogFlow | AI based image processing | AI solution providers in Bangalore | AI based Predictive Analytics | Conversational Bots Development in Bangalore | AI chatbots and voicebots | E-Commerce Solution Providers in Bangalore | Demandware Consulting Service in Bangalore | Demandware Companies in Bangalore | SFCC Consulting Service in Bangalore | SFCC Consulting Companies in Bangalore | SFCC Service Providers in Bangalore | Demandware Contract Staffing in Bangalore | Salesforce Commerce Cloud Consulting Services in Bangalore | SFCC Contract Staffing in Bangalore | Salesforce Commerce Cloud Contract Staffing in Bangalore | Oracle Consulting Services in Bangalore | Oracle Service Providers in Bangalore | Oracle Contract Staffing in Bangalore | OCC Contract Staffing in Bangalore | Oracle Commerce Cloud Consulting in Bangalore | Oracle Commerce Cloud Companies in Bangalore | SAP Hybris Consulting Services in Bangalore | SAP Hybris Service Providers in Bangalore | SAP Hybris Contract Staffing in Bangalore | SAP Hybris Commerce Cloud Consulting in Bangalore | SAP Hybris Companies in Bangalore | SAP Hybris Solutions in Bangalore | Hybris Commerce Solution in India | Hybris Solution Provider Companies | Magento Consulting Services in Bangalore | Magento Service Providers in Bangalore | Magento Contract Staffing in Bangalore | Magento Commerce Cloud Consulting in Bangalore | Magento Companies in Bangalore | Mobile App Development Company in Bangalore | Android App Development Services in Bangalore | Location Tracking Based Mobile App Development | Mobile App Development In Bangalore | Mobility Solution Provider in Bangalore | SQL Server Support Services in Bangalore | SQL Server Support Companies in Bangalore | Data Mining Solution in Bangalore | Custom App Development in Bangalore | Contract Staffing Solution in Bangalore

05SepSeptember 5, 2022

Orchestrating Multi-Brokers Kafka Cluster through CLI Commands

This short article aims to highlight the list of commands to manage a running multi-broker multi-topic Kafka cluster utilizing built-in scripts. These commands will be helpful/beneficial when the cluster is not integrated or hooked up with any third party administrative tool having GUI facilities to administer or control on the fly. Of course, most of them are not free to use. Can refer here to set up a multi-broker Kafka cluster.By executing the built-in scripts available inside the bin...

By Gautam GoswamiApache Kafka, Data EngineeringApache Kafka, Big Data, bin directory of the Kafka, Kafka, Kafka Administration, list of commands to run multi-broker multi-topic Kafka cluster, modify a Kafka topic, number of replication factor can’t be higher than the number of broker in the cluster, Orchestrating Multi-Brokers Kafka Cluster, Orchestrating Multi-Brokers Kafka Cluster through CLI Commands, retention time of a Kafka topic, retrieve log directories information from the kafka brokersComments Off

25AugAugust 25, 2022

Importance of Schema Registry on Kafka Based Data Streaming Pipelines

Needless to say Apache Kafka delivers messages to both real-time and batch consumers without performance degradation and in addition to that gaining enormous momentum as a foremost component for data streaming pipelines too. Credit card fraud detection, predictive maintenance, or real-time analytics, building streaming IoT platform, etc are the example of real-time use cases. To handle massive amounts of data ingestion, Apache Kafka is the cornerstone of a robust IoT data platform. A schema defines the structure of the data...

By Gautam GoswamiApache Kafka, Architecture, Data Engineering, Data IngestionApache Kafka, assign schema info in the schema registry, Avro, building streaming IoT platform, centralized schema management, Confluent Schema Registry, Credit card fraud detection, Data ingestion, data pipeline, Data Pipelines, Data Streaming, deserialized the messages, distributed storage layer for schemas, Hadoop, JSON Schema, Kafka based data pipeline, Kafka based data streaming pipelines, Kafka connect, Kafka producers and consumers, multi-broker Kafka topic, Multi-node kafka cluster, predictive maintenance, producer-consumer contract, Protobuf schemas, real-time analytics, schema change history, schema evolution, schema of registered data streams, schema registry, Schema Registry on Kafka Based Data Streaming Pipelines, Schema Registry on Kafka Streaming Pipelines, service layer for metadata, streaming applications, streaming data to Kafka topicComments Off

18AugAugust 18, 2022

Why Kappa Architecture for processing of streaming data. Have competence to superseding Lambda Architecture?

Data is quickly becoming the new currency of the digital economy, but it is useless if it can’t be processed. The processing of data is essential for subsequent decision-making or executable actions either by the human brain or various devices/applications etc. There are two primary ways of processing data namely batch processing and stream processing. Typically batch processing has been adopted for very large data sets and projects where there is a necessity for deeper data analysis, on the...

By Gautam GoswamiArchitecture, Data Engineering, Data IngestionAmazon Kinesis, Apache Flink, Apache Hadoop, Apache Kafka, Apache Samza, Apache Storm, batch processing layer, Big Data, Data Lake, data warehouse, event based Kappa architecture, event streaming platform, Hadoop Data Lake, HDFS, Kafka, Kafka Architecture Development Kafka Architecture Development, Kappa Architecture, Kappa Architecture for streaming data, Lambda Architecture, Map-Reduce framework, messaging engine, multiple stream processors, stream processing application, streaming computation system, streaming data analyticsComments Off

19JanJanuary 19, 2021

iDropper – The Data Ingestion, Monitoring and Reporting Tool

In today’s complicated world of business, the data, organizations own and how they use it, make them different from others to innovate, to compete better and to stay ahead in the business. That’s the driving factor for the organizations to collect and process as much data as possible, transform it into information with data-driven discoveries, and deliver it to the end user in the right format for smart decision-making. Common Challenges/Concerns Fetching the raw data files from the various data...

By Kislay KomalData Analysis, Data IngestionBig Data Analytics Big Data Analytics, Creating Data Lake Creating Data Lake, Data Absorption Software, Data Absorption Tool, Data Ingestion and Reporting, Data Ingestion and Reporting Software, Data Ingestion Software, Data Ingestion Tool, IDropper, Migrating data from multiple data sources, The Data Ingestion and Reporting SoftwareComments Off

11SepSeptember 11, 2020

Coming Along with Chat bots

Chatbot Service Moving your business where users use it most. Providing users access to information at their fingertips by where information is just a question away. Providing a conversation interface to users to query the information they need without having a human intervention. We can help you build Conversation AI driven chatbot which will help you provide a better user experience and expand your each to customers. Benefits of a Chatbot Availability A chatbot is available round the clock ensuring assistance is not...

By Kislay KomalArtificial Intelligence, ChatBots & VoiceBotsAlexa Skills Kit, Amazon Lex, Benefits of a Chatbot, Chatbot assitance for Corono, Chatbot for Covid-19, Chatbot platforms, Chatbot Service, common use cases of a Chatbot, Customer or Employee Self-service, DialogFLow, Employee Onboarding, FAQ Chatbot, Google Actions, Landbot, Quality Support Chatbot, Rasa, Reporting Chatbot, Store LocatorComments Off

11SepSeptember 11, 2020

Store Locator Bot

Problem Statement During the Covid-19 pandemic how will one know if stocks are available at a nearby retail store. As we all might have experienced that before or during the lock-down a few individuals purchased a lot of items in bulk and for oneself and family without considering the risk of stocks not being available to the rest. Also, to ensure each of us stay home and not go out every now and then to the stores we purchases groceries...

By Kislay KomalChatBots & VoiceBotsbenefits of bots, Chatbot for Whatsapp, Chatbot Using DialogFlow, Covid-19 pandemic, Covid-19 pandemic lock-down, Covid-19 Problem, Covid-19 solution, development of the chatbot, Locating Stores using bot, on the go assistance bots, Store Locator Bot, Store Locator ChatbotComments Off

18MayMay 18, 2020

Hiring During Covid-19

Dear Friends, At a time when there is a noise of layoffs and pay cuts, Irisidea decided to shun the negatives and look at the positives. Amid all the gloom surrounding the job market due to the COVID-19 pandemic, there’s good news. You can upgrade, up-skill and update your learning while getting adjusted to new normal. Not all companies are firing and laying of people. Some companies, including us, are still hiring. Apart from the positions open for senior technical resources,...

By Kislay KomalCovid-192020 hiring, AI jobs, Big Data & Analytics jobs, Chatbot Development jobs, covid, covid hiring, covid jobs, covid-19, E commerce Development jobs, fresh graduates hiring, hiring during covid, interns hiring, Irisidea, Machine Learning jobs, Marketing jobs, Mobile App Development jobs, UI-UX Designer jobs HR jobsComments Off

30MarMarch 30, 2019

Error while batch processing of rest data persisted in Basic Hadoop based (HDFS) Data Lake “Permission denied: user=dr.who, access=READ_EXECUTE, inode=”/tmp”:hdadmin:supergroup:drwx……..”

Typically, persisting unstructured data and subsequent batch processing can be very costly and is not advisable for small organizations & startups, as cost is prime factor for them. A Hadoop based Data Lake using Map-Reduce, fits perfectly in this scenario which is not only cost effective but also scalable and easy to extend further. Though it may sound a great option to have, we might face issues while setting up the same and one of common issues is, error "Permission...

By Kislay KomalData Engineering, Multi Node Hadoop Cluster Setup"Permission denied: user=dr.who, access=READ_EXECUTE, Error while batch processing of rest data persisted, Error while batch processing of rest data persisted in Basic Hadoop based (HDFS) Data Lake "Permission denied: user=dr.who, Error while batch processing of rest data persisted in Basic Hadoop based Data Lake, Error while batch processing of rest data persisted in Basic HDFS, Error while batch processing of rest data persisted in HDFS, inode="/tmp":hdadmin:supergroup:drwx........"Comments Off

29MarMarch 29, 2019

Data Analysis Vs Predictive Analysis Vs Artificial Intelligence

What is Artificisal Intelligence? In simple words, Artificial intelligence, sometimes called machine intelligence, is intelligence demonstrated by machines, to learn from experience, adjust to new inputs and perform human-like tasks. Data Analysis - Predictive Analysis – Artificial Intelligence Data analysis refers to reviewing data from past events for patterns. Predictive analysis is making assumptions and testing based on past data to predict future what/ifs. AI machine learning analyzes data, makes assumptions, learns and provides predictions at a scale and depth of detail...

By Kislay KomalArtificial IntelligenceAI machine learning analyzes data, Artificial Intelligence, Data Analysis, Data analysis refers to reviewing data from past events for patterns, Data Analysis Vs Predictive Analysis Vs Artificial Intelligence, learns and provides predictions, makes assumptions, Predictive Analysis, Predictive analysis is making assumptions and testing based on past data to predict future, What is Artificisal IntelligenceComments Off

27MarMarch 27, 2019

Resolved – ” Incompatible clusterIds in… ” in Multi Node Hadoop Cluster Setup

Currently, there are many startups / small companies and their customers, working on Data Analytics, ML, AI and related solutions. Due to their budget constraints, some of them don't want to leverage Cloud-based storage. Alternatively, to process ingested data, they create basic Data Lake using HDFS. During this process, they might encounter the exception of "org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /home/....". while starting the Name Node or Master Node in a multi-node Hadoop Cluster. This may occur in the following scenarios: ...

By Gautam GoswamiApache Hadoop, Data Engineering, Multi Node Hadoop Cluster Setup/home/hadoop/data/dataNode, > hdfs namenode -format, adding a new Data Node, adding a new Data Node to the cluster, Adding different operational node to cluster, Adding different operational node to cluster based on increasing data volume, clusterID, dfs.datanode.data.dir, dfs.namenode.name.dir, format the Name node, formatting the Name node, FsImage, FsImage file, HADOOP Cluster, Hadoop Cluster Setup, hdfs-site.xml, Incompatible clusterIds in, Multi Node Hadoop Cluster, Multi Node Hadoop Cluster Setup, Name Node and Master Node, org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /home/, Resolved Incompatible clusterIds in, setting up HDFS data lake for organizations, setting up HDFS data lake for startups, setting up your own HDFS data lake, Starting Master Node, Starting Name NodeComments Off