Data Engineering

Mobile Phone Authentication and Fraud

Day by day, we are getting addicted to the mobile phone especially smart phone since smart phone performs many of the functions of a computer, typically having a touch screen interface, Internet access. With the extensive growth of mobile applications, we can utilize various mobile applications starting from games to financial transaction including stock market brokerage. Many banks have launched their own mobile applications so that customer can download and start financial transactions like balance amount verification, money transfer...

Read more...

Basic concept of Data Lake

The left side info graphics represents the basic concept of Data Lake where we can use the approach of ELT (Extraction, loading and then transformation) against traditional ETL (Extraction, Transformation and then loading) process. ETL process implies to traditional data warehousing system where structured data format follows (row and column). By leveraging HDFS (Hadoop Distributed File System), we can develop data lake to store any format data in order to process and analysis. Directly data can be loaded in the Lake...

Read more...

Real time data analytics helps mobile service providers to achieve aggressive advantages

Usage of smart phones has become an integral part of our daily routine. Keeping aside phone calls and SMS, we are always engaged with lots of other activities Starting from entertainment to domestic shopping, social engagement etc., by installing various types of mobile applications. Of course, mobile internet is mandatory to carry out above.  Mobile service providers are facing new and difficult challenges. Due to exponential growth of customer's expectations, they need to serve accordingly with advanced mobile technology and handle...

Read more...

How Google news is able to group similar news together

Google news uses clustering machine learning techniques to group similar kind of news or articles together.  Interestingly, they don't have thousand news editors on trunk instead use the clustering techniques to forms groups of similar data based on the common characteristics. Mahout is a machine learning software from Apache community that applications leverage to analyse large sets of data.  Before invention of Mahout, it was too complex to a analyse large sets of data. Mahout extensively utilize Apache Hadoop to...

Read more...

Essentially of Data Wrangling

To roll out a new software product commercially irrespective of any domain in the market,  360-degree quality check with test data is mandatory.  We can correlate this with a visualized concept of a new vehicle.  After completion of vehicle manufacturing, fuel has to be injected to the engine to make it operational. Once the vehicle starts moving, all the quality checks, testing get started like brake performance, mileage, comfort etc with thousands of other factors which are decided/concluded during...

Read more...

Semi-Structured Data

Semi-structured data lies between structured and unstructured data. Data that get stored in the traditional database system or excel sheet can be denoted as structured data and organized in COLUMNS and ROWS. Unstructured data can be considered as any data or piece of information which can't be stored in Databases/RDBMS etc. Email, Facebook comments, news paper etc. are the examples of unstructured data. Semi-structured data do not follow strict data model structure and neither raw data nor typed data in...

Read more...

Big Data Generation and its sources

Here are the sources where we can visualized how Big data is generating today and anticipate how the entire globe sink under Oceans of Data in near future due to exponential growth of digitization. Media Media and communications outlet (article, audio, video, emails, blogs, podcasts) Machine Data generated by computers and machines generally without human intervention (Server logs, phone calls, sensors, business process logs etc). Social Digital materials created by social media like Face book, LinkedIn etc (texts, photos, videos, tweet etc) Historical Data about our environment (weather, traffic,...

Read more...

Data in Motion

The information security strategy in today's business world is important and critical. Information is nothing but data in various shape like flat file, audio, video files, multimedia etc.  The data always lays in three states. Data in Rest This state we can visualize when information or data stores in Databases (RDBMS), Hard disk, Pen drive, SD card in smart phone etc. Organizations are bound to have additional layers as defense to protect sensitive data from intruders in the event that the...

Read more...

Importance of unstructured data

In today's world,  Internet plays a major factor to generate and propagate information from various sources. Social media, Email, What'sApp, E-News Paper  etc  are playing a crucial role on circulation followed by creation of information.  These type of information often include text and multimedia contents.  These information or data methodically can't be persisted in database and it is referred as unstructured data. Due to advancement in  technology,  70-80 % growing data is unstructured and increasing significantly over structured/semi structured data....

Read more...

Hadoop Development Environment

We always talk about Big Data processing using Hadoop framework. By leveraging distributed cluster computing programming, now a days it is possible to process and analysis huge volume of data probably in exabytes or more than that. Giant cloud providers like Amazon, Microsoft Azure provides hosting as well as development environment with multi-node cluster according to hardware requirements on Pay Per Use model.  They usually charge on hourly basis. But the main concern is, to open an account a...

Read more...