Author - Gautam Goswami

04Feb

Importance of unstructured data

In today's world,  Internet plays a major factor to generate and propagate information from various sources. Social media, Email, What'sApp, E-News Paper  etc  are playing a crucial role on circulation followed by creation of information.  These type of information often include text and multimedia contents.  These information or data methodically can't be persisted in database and it is referred as unstructured data. Due to advancement in  technology,  70-80 % growing data is unstructured and increasing significantly over structured/semi structured data....

By Apache Hadoop, Data EngineeringComments Off
Read more...
04Feb

Hadoop Development Environment

We always talk about Big Data processing using Hadoop framework. By leveraging distributed cluster computing programming, now a days it is possible to process and analysis huge volume of data probably in exabytes or more than that. Giant cloud providers like Amazon, Microsoft Azure provides hosting as well as development environment with multi-node cluster according to hardware requirements on Pay Per Use model.  They usually charge on hourly basis. But the main concern is, to open an account a...

By Apache Hadoop, Data EngineeringComments Off
Read more...
18Jan

Technology Platform behind Aadhaar card implementation

We are almost familiar with Aadhaar card which had been rolled out as a first initiative in 2003. It's a 12-digit unique identification number issued by the Indian government to every individual resident of India and that can be used to access a variety of services and benefits. Hadoop is an open-source big data processing framework, that has been customized excessively by the company named MapR to boost performance.The aadhaar card project has been developed using MapR's customized Hadoop and...

By Apache Hadoop, Data Engineering, Hadoop Eco SystemComments Off
Read more...
18Jan

Challenges in data analysis due to Demonetization

We are all aware of demonetization and its ongoing impact in the entire country. Government is trying hard to digitalize entire financial system. In nutshell, initially to create lower cash transaction before reaching to complete cashless transaction. Use of different types digital wallet, mobile banking, debit and credit cards are started booming among the people of the country and citizens slowly started adopting it mainly to get rid of long queues in ATM, Banks and due to their advantages...

By Data EngineeringComments Off
Read more...
18Jan

Introduction to Facebook graph Application Program Interface( API).

API is a consolidation of protocols, set of routines and tools for developing software application. Ideally Facebook recognizes the relationship among the entities as a "Social Graph". In Facebook, an entity can be a person, place, event or object that is relevant to a given system and an attribute is a property , characteristic of the entity. As shown in the diagram, Location is an entity and country, state, location id are the attributes. Based on the registered user...

By Data EngineeringComments Off
Read more...
18Jan

DYNAMICS OF ONLINE SOCIAL NETWORKS

Mathematical models for the analysis of spread of social influence have emerged as a major topic of interest among the researchers in diverse disciplines such as sociology, economics and computer science. Empirical studies of diffusion on social networks date back to the 1940s. Later on, theoretical propagation models were introduced in late 1970s. In the recent years, Online Social Networks such as Facebook, Twitter and Linkedin has experienced explosive growth and these have remarkably changed the way people communicate....

By Data EngineeringComments Off
Read more...
18Jan

Data storage mechanism in Facebook

  We are all almost familiar with social media mainly Facebook where photo uploads total 300 million per day. Daily generates 4.5 billion likes. Every 60 seconds, 510 comments are posted, 293,000 statuses are updated. It's really curious to know how Facebook stores such a huge volume of data that's totally impossible using traditional database management systems (RDBMS) . Facebook is using distributed database management system called Cassandra. Cassandra was initially developed by Facebook to enhance their inbox search feature. In...

By Data EngineeringComments Off
Read more...
18Jan

Mainframe Applications slowly migrating to Hadoop

The giant organizations across the globe are using legacy mainframe systems due to it's scalability, security and reliability of machine's processing capacity subjected to heavy and large workloads. Of course, these infrastructures desire huge hardware, software and processing capacity. As the technology advancing very rapidly, scarcity of mainframe technicians, developers etc are increasing and it has become a major challenge for those organizations to continue their operations. The maintenance/replacement of these hardware are also another threat due to low...

By Apache Hadoop, Data Engineering, Hadoop Eco SystemComments Off
Read more...
18Jan

How Social Media Making Irritating Login Process Comfortable !

  Nowadays social identities are becoming an integral part of a web presence that deals with different activities. With the advancement of technology, most of the traditional transactional activities have been lifted to the web like shopping in e-commerce portal, Travel booking for Bus, Train, Flight, Hotel reservation in holiday destination etc. Merchandisers involve in above businesses, host their own web portal to facilitate the services. When a visitor browses to a web portal, it is mandatory to create his new...

By EcommerceComments Off
Read more...
18Jan

Establishment of Data Lake specific to multi-channel e-commerce application to understand customer’s buying pattern

Post order fulfillment data is becoming a very important asset of e-commerce vendors to understand complete buying pattern of customers. Especially for the e-commerce vendors who sells multiple products starting from electronics to apparels. Extraction and transformation are time-consuming operations when partially structured data starts moving from the various sources and finally land into the relational data warehouse.  Data extracted from the social media are semi-structured (JSON or XML).  As an example, Facebook provides information in JSON format through Graph API and same...

By Apache Hadoop, Data Engineering, Data Ingestion, Hadoop Eco System, Processing Engine, Storage MechanismComments Off
Read more...