Yearly Archives - 2017

17Jan

Information analysis using Hadoop

After Kerala's Puttingal Devi Temple fire tragedy, we can visualize sudden data explosion in all digital media. After that tragic incident, huge amount of data are generated in the form of text, voice, photo, video, blogs etc. over internet via social media, news channels, e-news papers and comments, sentiments, various opinions are flooded on whether fire crackers burst should be allowed in devotional places or not. This is a classic example of Big Data where existing traditional softwares are incapable...

By Data EngineeringComments Off
Read more...
17Jan

Big data approach in Banking system

Typically Banking systems are responsible to validate and verify financial transaction data, geo-location data from mobile devices, merchant data, and authorization including submission data. Data from lots of social media channels and Banking’s mainframe data center have a significant challenge to process and deliver final output. Issue:- Legacy systems are incapable of processing the data in when is in motion. Combining all different format of data is together is another challenge like structured, semi- structured and un-structured. Big data Approach:- Big data analytics enables to...

By Data EngineeringComments Off
Read more...
17Jan

Performance of Hadoop Map-Reduce

The performance of Hadoop Map-Reduce job can be increased amicably without investing more on the hardware cost. Simply tuning some parameters according to the cluster specifications, input data size and processing complexities. Here are few general tips to improve Map_reduce job performance - Always we should use compression when writing intermediate data (mapper output) to disk before shuffling - Include combiner in the appropriate position. - LongWritable data type is incorrect as output when range of output values are in Integer range. IntWritable...

By Architecture, Data EngineeringComments Off
Read more...