Data Governance & Security Mechanism in Distributed Data Storage System
We are aware that the traditional data storage mechanism is incapable to hold the massive volume of data generated with lightning speed for further utilization even if we perform vertical scaling, and we have anticipated only one fuel, nothing but DATA to accelerate the movement across all the sectors starting from business to natural resources including medical towards rapid growth.
But the question is how to persist this massive volume of data for processing? The answer is, storing the data in a distributed multi-node cluster environment, where it can be scaled linearly on demand. The former statement is made physically achievable by Hadoop distributed file system (HDFS). Using HDFS we can store data in a distributed manner (multi-node cluster where the number of nodes can be increased in the cluster linearly as data grows). Using Hive and HBase we can organize the HDFS data and make it more meaningful as the data become queryable. The next hurdle in the path of accelerating the movement towards growth, is to govern the data and security implication on this huge volume of persisted data.
In a single statement, data governance can be defined as the consolidation of managing data access, accountability and security. By default, HDFS does not provide any strong security mechanism to achieve complete governance but with the additional combination to the following approach, we can proceed towards it.
- Integration with LDAP – To secure read/write operation on the persisted data, appropriate authorization with proper authentication is mandatory. Authentication can be achieved in HDFS by integration with LDAP server across all the nodes. LDAP is often used as a central repository for user information and as an authentication service. Organization/Company who has ingested huge data into Hadoop for analysis can define the security policy to avoid data theft, leak, misuse and ensure the right access to data inside HDFS directories, execute HIVE query etc. The user or team need to get authenticated via LDAP server before processing/query data from the cluster. LDAP integration with Hadoop can be done either by using OS-level configuration to read LDAP groups or explicitly configuring Hadoop to use LDAP-based group mapping.
- Introducing Apache Knox gateway – Single access point with multi-node Hadoop clusters can be achieved by Apache Knox for all REST and HTTP interactions. The complex configuration, the client-side library can be wiped out by using Apache Knox. Besides accessing data in the cluster, we can provide security for job execution in the cluster.
- Kerberos for authentication – Kerberos network authentication protocol provides strong authentication for the 2-tier application (client and server). Kerberos server verifies identities for every request when the client wants to access Hadoop cluster. Kerberos Database stores and controls all principles and realms. Kerberos uses secret-key cryptography to enhance strong authentication by providing user-to-server authentication. A Kerberos server, usually called Key Distribution Center (KDC) should be installed on one physical host and it’s database contains the user and service entries like user’s principal, maximum validity, maximum renewal time, password expiration, etc.
By Gautam Goswami
