Apache Hadoop - Real-time Data Streaming, Real-time Data Processing, Real-time Data Analytics | Data Engineering Solution in Bangalore | Apache Kafka Streaming Solutions in Bangalore | Kafka Confluent Cloud Solutions in Bangalore | Kafka Streaming Implementation Support in Bangalore | Apache Kafka Support in Bangalore | Multinode Kafka Cluster Setup in Bangalore | Kafka Application Consulting in Bangalore | Kafka cloud implementation in Bangalore | Kafka infrastructure consulting in Bangalore | Kafka security implementation in Bangalore | Kafka upgrade support in Bangalore | Zookeeper setup support in Bangalore | Zookeeper Solutions in Bangalore | Multinode Zookeeper Setup in Bangalore | Big Data Consulting Service Providers in Bangalore | Data Analytics Consutling Services in Bangalore | Big Data Solution Providers in Bangalore | Big Data Analytics Companies in Bangalore | Data Analytic Services in Bangalore | Big Data Services in Bangalore | Big Data Analytics Solutions in Bangalore | Big Data Analytics Service Providers in Bangalore | Big Data Case Studies | Big Data Companies in Bangalore | Multi Node Hadoop Cluster | Data Lake creation and support | Data Ingestion Services in Bangalore | Koolanch | Artificial Intelliegence Solutions in Bangalore | Predictive Analysis Solution in Bangalore | Machine Learning Solution in Bangalore | Deep Learning Solutions Bangalore | ChatBots for Websites | Text to Speech API | DialogFlow ChatBots | ChatBots using DialogFlow | AI based image processing | AI solution providers in Bangalore | AI based Predictive Analytics | Conversational Bots Development in Bangalore | AI chatbots and voicebots | E-Commerce Solution Providers in Bangalore | Demandware Consulting Service in Bangalore | Demandware Companies in Bangalore | SFCC Consulting Service in Bangalore | SFCC Consulting Companies in Bangalore | SFCC Service Providers in Bangalore | Demandware Contract Staffing in Bangalore | Salesforce Commerce Cloud Consulting Services in Bangalore | SFCC Contract Staffing in Bangalore | Salesforce Commerce Cloud Contract Staffing in Bangalore | Oracle Consulting Services in Bangalore | Oracle Service Providers in Bangalore | Oracle Contract Staffing in Bangalore | OCC Contract Staffing in Bangalore | Oracle Commerce Cloud Consulting in Bangalore | Oracle Commerce Cloud Companies in Bangalore | SAP Hybris Consulting Services in Bangalore | SAP Hybris Service Providers in Bangalore | SAP Hybris Contract Staffing in Bangalore | SAP Hybris Commerce Cloud Consulting in Bangalore | SAP Hybris Companies in Bangalore | SAP Hybris Solutions in Bangalore | Hybris Commerce Solution in India | Hybris Solution Provider Companies | Magento Consulting Services in Bangalore | Magento Service Providers in Bangalore | Magento Contract Staffing in Bangalore | Magento Commerce Cloud Consulting in Bangalore | Magento Companies in Bangalore | Mobile App Development Company in Bangalore | Android App Development Services in Bangalore | Location Tracking Based Mobile App Development | Mobile App Development In Bangalore | Mobility Solution Provider in Bangalore | SQL Server Support Services in Bangalore | SQL Server Support Companies in Bangalore | Data Mining Solution in Bangalore | Custom App Development in Bangalore

06JanJanuary 6, 2023

Few intrinsic of Apache Zookeeper and their importance

As a bird’s eye view, Apache Zookeeper has been leveraged to get coordination services for managing distributed applications. Holds responsibility for providing configuration information, naming, synchronization, and group services over large clusters in distributed systems. To consider as an example, Apache Kafka uses Zookeeper for choosing their leader node for the topic partitions. Please click here if you want read on how to setup the multi-node Apache Zookeeper cluster on Ubuntu/Linux zNodes The key concept of the Zookeeper is the znode which can be acted...

By Gautam GoswamiApache Hadoop, Apache Kafka, Architecture, Data Engineering, Hadoop Eco SystemApache Kafka, Apache Zookeeper, Big Data, Concept of zNodes, data log directory, dataDir parameter, dataLogDir parameter, distributed file systems, ephemeral zNodes, Hadoop, how to setup the multi-node Apache Zookeeper, Importance of Apache Zookeeper, persistence zNodes, receive notifications about changes to the ZooKeeper ensemble through watches, sequentialzNodes, Usage of Apache Zookeeper in Kafka Multi Node CLuster, Using Apache Zookeeper for managing distributed applications, ZooKeeper Data Directory, ZooKeeper ensemble, Zookeeper QuorumComments Off

27MarMarch 27, 2019

Resolved – ” Incompatible clusterIds in… ” in Multi Node Hadoop Cluster Setup

Currently, there are many startups / small companies and their customers, working on Data Analytics, ML, AI and related solutions. Due to their budget constraints, some of them don't want to leverage Cloud-based storage. Alternatively, to process ingested data, they create basic Data Lake using HDFS. During this process, they might encounter the exception of "org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /home/....". while starting the Name Node or Master Node in a multi-node Hadoop Cluster. This may occur in the following scenarios: ...

By Gautam GoswamiApache Hadoop, Data Engineering, Multi Node Hadoop Cluster Setup/home/hadoop/data/dataNode, > hdfs namenode -format, adding a new Data Node, adding a new Data Node to the cluster, Adding different operational node to cluster, Adding different operational node to cluster based on increasing data volume, clusterID, dfs.datanode.data.dir, dfs.namenode.name.dir, format the Name node, formatting the Name node, FsImage, FsImage file, HADOOP Cluster, Hadoop Cluster Setup, hdfs-site.xml, Incompatible clusterIds in, Multi Node Hadoop Cluster, Multi Node Hadoop Cluster Setup, Name Node and Master Node, org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /home/, Resolved Incompatible clusterIds in, setting up HDFS data lake for organizations, setting up HDFS data lake for startups, setting up your own HDFS data lake, Starting Master Node, Starting Name NodeComments Off

02MarMarch 2, 2019

Network Topology To Create Multi Node Hybrid Cluster For Hadoop Installation

The aim of this article is to provide an outline for creating network topology for Hadoop installation in multi node hybrid cluster with limited available hardware resources. This cluster would be beneficial for learning Hadoop, with lower volume of unstructured data processing using various engines etc. Before the cluster setup: We installed Hadoop on a single node cluster running on Ubuntu 14.04 on top of Windows 10 using VMware workstation player. Later we have copied the .vmx file into multiple...

By Gautam GoswamiApache Hadoop, Data Engineering.vmx file, Assign static IP to Ubuntu VMware workstation players, Choosing a network switch over router, Configuration of data nodes with name node, Configure Internet Protocol Version 4 (TCP/IPv4) on Windows, Create Multi Node Hybrid Cluster, Create Multi Node Hybrid Cluster For Hadoop Installation, creating network topology for Hadoop installation, D-Link DES-1005C 10/100 Network Switch, Dell Inspiron 1525 Laptop with 2 GB RAM and Ubuntu 14.04 as operating system, Dell Inspiron 5458 Laptop with 16 GB RAM and Windows 10 as host operating system, Desktop with 8 GB RAM and Windows 7 Professional as host operating system, Ethernet LAN setup using Network Switch, guest OS, Hadoop, Hadoop Installation, Lenovo B40-80 Laptop with 4 GB RAM and Windows 10 as host operating system, Multi Node Hybrid Cluster, Multi Node Hybrid Cluster For Hadoop Installation, Network Adapter setting on the VM Player, Network Topology For Hadoop Installation, Network Topology To Create Multi Node Hybrid Cluster, Network Topology To Create Multi Node Hybrid Cluster For Hadoop Installation, OS Ubuntu 14.04, PC to Router, PC to Switch, Router to Switch, single node cluster running on Ubuntu, Straight-through cables, Ubuntu on top of Windows, Ubuntu on top of Windows 10 using VMware workstation player, VMware Workstation 12 player, VMware Workstation 7.x player, Windows using VMware workstation playerComments Off

25JunJune 25, 2018

Data Governance & Security Mechanism in Distributed Data Storage System

We are aware that the traditional data storage mechanism is incapable to hold the massive volume of data generated with lightning speed for further utilization even if we perform vertical scaling, and we have anticipated only one fuel, nothing but DATA to accelerate the movement across all the sectors starting from business to natural resources including medical towards rapid growth. But the question is how to persist this massive volume of data for processing? The answer is, storing the data...

By Gautam GoswamiApache Hadoop, Data Engineering, Storage MechanismApache Knox gateway, Data Governance, Data Governance & Security Mechanism in Distributed Data Storage System, Data Security Mechanism, Data Security Mechanism in Distributed Data Storage System, Data Storage System, Distributed Data Storage System, HDFS, HDFS cluster, Integration with LDAP, Kerberos, Kerberos for authentication, managing data accessComments Off

02JunJune 2, 2018

Processing and Analysis of Big Telecom Data to minimize crime, combat terrorism, unsocial activities etc.

Telecom providers have a treasure trove of captive data - customer data, CDR (call detail records), call center interactions, tower logs etc. and are metaphorically “sitting on a gold mine”. Ideally, each category of the generated data has the following information. ⦁ Customer data consolidates customer id, plan details, demographic, subscribed services and spending patterns ⦁ Service data category consolidates types of customer, customer history, complain category, query resolved etc. are on ⦁ Usually for the smart mobile phone subscriber,...

By Kislay KomalApache Hadoop, Architecture, Data Analysis, Data EngineeringBig Data Processing and Analysis, Processing and Analysis of Big Telecom Data, Processing and Analysis of Telecom Data, Telecom Data, Telecom Data Processing and AnalysisComments Off

21JanJanuary 21, 2018

Deleting Solr log files/folder from Standby NameNode could be the disaster when Primary NameNode is active in the HDP (Hortonworks Data Platform) Hadoop Cluster

Most of us know that we use Apache Ambari for managing, provisioning and monitor different components of a Hortonworks Hadoop cluster. We also know that Apache Ranger can be used as a centralized security administration solution for Hadoop that enables administrators to create and enforce security policies for HDFS and other Hadoop platform components. When ranger hdfs plugin is enabled ,it writes the client interaction activity to Solr if it is configured. The default location of this solr log files...

By Kislay KomalApache Hadoop, Data EngineeringComments Off

12DecDecember 12, 2017

Fault Tolerance Enhancement On Apache Hadoop 3.0.0-alpha2 For Supporting More Than 2 NameNodes

NameNode is the most critical resource in Hadoop core cluster. Once very large files loaded into the Hadoop Distributed File System (HDFS), the files get broken into block-sized chunks as per the parameter configured (64 MB by default). The chunks are then stored as independent units across the data nodes in the cluster. The primary responsibility of the data nodes is to hold the actual data in the form of chunk and NameNode holds the information where all the chunks located/stored in the...

By Kislay KomalApache Hadoopactive NameNode, Apache Hadoop, Apache Hadoop 3.0.0, Apache Hadoop 3.0.0-alpha2, configuration of NameNodes, data nodes, data nodes location, Fault Tolerance Enhancement On Apache Hadoop, filesystem metadata, filesystem namespace, filesystem tree, FsImage, FsImage file, Hadoop 2.0.0, Hadoop 3.0.0, Hadoop core cluster, Hadoop Distributed File System, Hadoop framework, HDFS, HDFS cluster, HDFS NameNode, JournalNodes, Master Node, NameNode, NameNode holds the information, new features of Apache Hadoop 3.0.0, new features of Hadoop 3.0.0, Primary NameNode, Quorum-based Storage, responsibility of the data nodes, secondary NameNode, single point of failure, SPOF, standby NameNode, Supporting More Than 2 NameNodes, very large amount of enterprise dataComments Off

14FebFebruary 14, 2017

Semi-Structured Data

Semi-structured data lies between structured and unstructured data. Data that get stored in the traditional database system or excel sheet can be denoted as structured data and organized in COLUMNS and ROWS. Unstructured data can be considered as any data or piece of information which can't be stored in Databases/RDBMS etc. Email, Facebook comments, news paper etc. are the examples of unstructured data. Semi-structured data do not follow strict data model structure and neither raw data nor typed data in...

By Gautam GoswamiApache Hadoop, Data EngineeringAnalysing unstructured Data, bi-directional data interchange, Cassandra, client-server web application, Email, excel sheet, Facebook comments, Facebook graph, Facebook graph API, GET method, GET method in REST service, graph API, Hadoop Distributed File System, HBase, HDFS, JavaScript object notation, JSON format, JSON or XML, MongoDb, news paper, NoSQL, NoSQL Database, REST service, REST service request/response, sample unstructured data, Semi-structured data. structured data, traditional database system, transmit over wireComments Off

04FebFebruary 4, 2017

Importance of unstructured data

In today's world, Internet plays a major factor to generate and propagate information from various sources. Social media, Email, What'sApp, E-News Paper etc are playing a crucial role on circulation followed by creation of information. These type of information often include text and multimedia contents. These information or data methodically can't be persisted in database and it is referred as unstructured data. Due to advancement in technology, 70-80 % growing data is unstructured and increasing significantly over structured/semi structured data....

By Gautam GoswamiApache Hadoop, Data EngineeringComments Off

04FebFebruary 4, 2017

Hadoop Development Environment

We always talk about Big Data processing using Hadoop framework. By leveraging distributed cluster computing programming, now a days it is possible to process and analysis huge volume of data probably in exabytes or more than that. Giant cloud providers like Amazon, Microsoft Azure provides hosting as well as development environment with multi-node cluster according to hardware requirements on Pay Per Use model. They usually charge on hourly basis. But the main concern is, to open an account a...

By Gautam GoswamiApache Hadoop, Data EngineeringComments Off