Blog

Data Analysis Vs Predictive Analysis Vs Artificial Intelligence

What is Artificisal Intelligence? In simple words, Artificial intelligence, sometimes called machine intelligence, is intelligence demonstrated by machines, to learn from experience, adjust to new inputs and perform human-like tasks. Data Analysis - Predictive Analysis – Artificial Intelligence Data analysis refers to reviewing data from past events for patterns. Predictive analysis is making assumptions and testing based on past data to predict future what/ifs. AI machine learning analyzes data, makes assumptions, learns and provides predictions at a scale and depth of detail...

Read more...

Resolved – ” Incompatible clusterIds in… ” in Multi Node Hadoop Cluster Setup

Currently, there are many startups / small companies and their customers, working on Data Analytics, ML, AI and related solutions. Due to their budget constraints, some of them don't want to leverage Cloud-based storage.  Alternatively, to process ingested data, they create basic Data Lake using HDFS. During this process, they might encounter the exception of "org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException:  Incompatible  clusterIDs in /home/....". while starting the Name Node or Master Node in a  multi-node Hadoop Cluster. This may occur in the following scenarios: ...

Read more...

Network Topology To Create Multi Node Hybrid Cluster For Hadoop Installation

The aim of this article is to provide an outline for creating network topology for Hadoop installation in multi node hybrid cluster with limited available hardware resources.  This cluster would be beneficial for learning Hadoop, with lower volume of unstructured data processing using various engines etc. Before the cluster setup: We installed Hadoop on a single node cluster running on Ubuntu 14.04 on top of Windows 10 using VMware workstation player. Later we have copied the .vmx file into multiple...

Read more...

The Brain and Convolutional Neural Network (CNN)

In the beginning of our life parents/family used to tell us name of the objects we see. We learned by examples given to us. Slowly we started recognizing certain things more often. They became so common that next time we saw them we would instantly know the name of the object and details of that. We label every object based on what we have learnt in the past. This is what we subconsciously do all days, we see, label, make...

Read more...

Data Governance & Security Mechanism in Distributed Data Storage System

We are aware that the traditional data storage mechanism is incapable to hold the massive volume of  data generated with lightning speed for further utilization even if we perform vertical scaling,  and we have anticipated only one fuel, nothing but DATA to accelerate the movement across all the sectors starting from business to natural resources including medical towards rapid growth. But the question is how to persist this massive volume of data for processing? The answer is, storing the data...

Read more...

Processing and Analysis of Big Telecom Data to minimize crime, combat terrorism, unsocial activities etc.

Telecom providers have a treasure trove of captive data - customer data, CDR (call detail records), call center interactions, tower logs etc. and are metaphorically “sitting on a gold mine”. Ideally, each category of the generated data has the following information. ⦁ Customer data consolidates customer id, plan details, demographic, subscribed services and spending patterns ⦁ Service data category consolidates types of customer, customer history, complain category, query resolved etc.       are on ⦁ Usually for the smart mobile phone subscriber,...

Read more...

Deleting Solr log files/folder from Standby NameNode could be the disaster when Primary NameNode is active in the HDP (Hortonworks Data Platform) Hadoop Cluster

Most of us know that we use Apache Ambari for managing, provisioning and monitor different components of a Hortonworks Hadoop cluster. We also know that Apache Ranger can be used as a centralized security administration solution for Hadoop that enables administrators to create and enforce security policies for HDFS and other Hadoop platform components. When ranger hdfs plugin is enabled ,it writes the client interaction activity to Solr if it is configured. The default location of this solr log files...

Read more...

Basic Understanding Of Stateful Data Streaming Supported By Apache Flink

Technologies related to Big Data processing platform are enhancing the maturity in order to efficiently execute the streaming data which is becoming a major focus point to take business decision instantly specially in telecom and retail sector. Collecting data continuously from the various sensors installed/fitted with an industrial heavy equipment, click stream on an e-commerce application’s navigation etc can be considered as streaming data generation sources. By leveraging streaming application, we can process/analyze these continues flow of data without...

Read more...

Apache Flink – A 4G Data Processing Engine

Analyzing streaming data in large-scale systems is becoming a focal point day by day to take accurate business decisions due to mushrooming of digital data generation sources around the globe including social media. Real-Time analytics are becoming more attractive due to possibilities of getting insights from the time-value of data (in other words, when data is in motion). Apache Flink, an open source highly innovative stream processor engine has been grounded which helps to take advantage of stream-based approaches. Besides...

Read more...