Data governance and (hence) Metadata Management

Data is a strategic asset for businesses today and it is growing exponentially. Regulations like GDPR, BCBS 239, Basel 3 are making it extremely important for organisations to have full control over their data. Data governance is about availability, usability, integrity and security of data.

Is this something new we are learning here? Actually, No! Data governance has been in place since years. Every system has some or other form of data governance in place. Documents and excel sheets are traditional form of data governance. Could be access control mechanism implemented at system level. Operational fields in a databases like created by/ updated by, are also part of this governance. Audit logs provides useful information for governance purposes. 

But is this sufficient? We are implementing new systems, new functionalities every day. Is the organisation working in silos where individual departments might have different standards? Is managing metadata centrally in excel sheets for huge enterprises a possibility? Do we have an enterprise wide agreed framework? Is the process repeatable? Will we be able to demonstrate our organisation’s data compliance when needed? 
These questions makes us think about data governance more seriously.

The problem with DG programmes is they sound too academic. What’s the ROI they bring in?! Well, the ROI is compliance. And process efficiencies. And it is going help businesses in short term and even more in long term. 
Involvement of all people, all departments is also critical for the success of a DG programme. Hence, educating people, involving them from the start is important. Breaking the silos is the key for a DG programme. 
Starting small with a specific objective is a good way to start. Though we start small, we need to keep an eye on the long term goal as well. 

There are multiple business and technology focus areas for DG. Like, process standardisation from business perspective.
Here we will talk about Metadata Management and Data Quality. I see these two as pillars of DG implementation. 
In wide enterprises, it is going to be extremely difficult to know what data you have and how to access it. There are many metadata management tools available in the market. Some are even open sourced. These tools not only capture the metadata automatically, but they also provide lineage, relationships, description, usage and ownership. Some tools even have built in machine learning which helps to categorise the data automatically. The tools are intuitive, user friendly, with Google like search functionality, which allows users to search by technical metadata or business term or even tag name of the field. Imagine, when we are designing new APIs, we won’t have to struggle in understanding what data we have and where to find it!
Built in business glossaries integrate the business and IT and allows them to work together.

Data quality and maintaining the customer consents globally for marketing and other purposes  is also a part of DG. A good DQ design will help in data completeness, accuracy, integrity, identifying and merging duplicates and standardising the data.


Implementing (and executing!) data governance practices will help businesses in becoming data driven. Organisations can become more compliant and be able to demonstrate the process, data security and access control.

Comments

Post a Comment

Popular posts from this blog

Understanding Wide-Column Stores

To understand the concept  more clearly, let’s start with an example of  Relational database : Customers Id Product FN Date Country 1 Dell Harry 17/04/2018 India 2 Dell Harry 17/04/2018 India 3 Apple Ron 17/04/2018 India 4 Sony Ron 17/04/2018 South Africa 5 Sony Hermione 17/04/2018 South Africa When we convert this to a  Column Store , this is how it looks like: FN LN Product Date Country Id Value Id Value Id Value Id Value Id Value 1-2 Harry 1-2 Potter 1-2 Dell 1-5 17/04/2018 1-3 India 3-4 Ron 3-4 Weasley 3 Apple 4-5 South Africa 5 Hermione 5 Granger 4-5 Sony Now a  Wide Column store  will Group the columns which are accessed frequently together into one , like Id Name 1-2 FN Harry LN Potter 3-4 FN Ron LN Weasley 5 FN Hermione LN Granger Some key points are, Columns
Read more

Two pizza team!

Image
Came across this terminology while reading about APIs based on microservices architecture. So, what are two pizza teams? Yes, they are small enough to be fed by 2 large pizzas! But how are they related to microservices?! We will find out. Amazon, being a popular online store, wanted to be able to innovate in IT space quickly. But their monolith architecture was not allowing them to make software changes quickly. Monolith architectures are the old school enterprise architectures where all the components of the software are tightly coupled. These APIs are powerful and perform lot of disparate functions. Even a small change in such systems needs a lot of planning, impact analysis, availability from other teams. This definitely slows down the software release cycle. Typically in such environments there are different teams like test team, deployment team, UI team, backend development team etc who work in silos. Any small or big change goes through a long process of build, test, deploy
Read more

Relational and Non-relational databases

In old days, when we used to think of 'database', we would only picture tables and columns, primary and foreign keys, complex joins, intersection tables etc. We often use ER diagrams, relationships to design and define such models. These databases are widely used and are stable. They adhere to  ACID  principles, provide  immediate consistency  which are most important when it comes to transaction handling. The most widely used products are  Oracle, Teradata, SAP Hana, IBM db2, MySQL, Amazon Aurora, Maria db (comes by default on linux distributions), postgreSQL . Then why is there a need of non-relational database? The reason is Data itself. Data is now considered most valuable resource, even more than Oil!! No wonder that social networking site like Facebook and twitter actually run the world. They decide the trends, what people read, what they think. All from the power of Data. Anyways that’s a topic for another post. Data from web, genomics data, data from IoT devices
Read more