I have been thinking about data governance, Azure Purview and data research. How these areas link together and enable innovation, that democratisation of data and holistically thinking about data as a strategic asset. A combination of above in my diagram covers my thoughts.
Starting with Azure Purview requires a few prerequisites. The checklist sets out 4 phases.
Identify where to start to establish a Governance baseline foundation for your organization for general cloud governance.
Azure Purview Readiness Checklist
Purview-Deployment-Checklist will help prepare for data governance and data democratization in your environment.
There are four sections
The Azure Purview deployment prerequisites can help with those proof of concept deployments
Azure Purview automated readiness checklist
A set of
scripts have been written to help evaluate your exciting environment for
missing configuration that might prevent data sources being scanned. The
PowerShell scripts are
Data Catalogs are becoming an essential component in the new data world. They are an inventory of an organisations data assets. The meta data collected helps in finding the most appropriate data at speed, know what data is held, the security levels and lineage of data. Microsoft have tool in preview that helps with data governance called Azure Purview.
Azure
Purview is a unified data governance service that helps you manage and govern
your on-premises, multicloud and software-as-a-service (SaaS) data. Easily
create a holistic, up-to-date map of your data landscape with automated data
discovery, sensitive data classification and end-to-end data lineage. Empower
data consumers to find valuable, trustworthy data.
Once
installed it moves into the hands of data governors. There are predefined data
plane roles dictating who can access what.
Purview
Data Reader - access to the Purview portal and can
read all content except for scan bindings
Purview Data
Curator - access to the
Purview portal and can read all content except for scan bindings, can edit
information about assets, classification definitions and glossary terms, and
can apply classifications and glossary terms to assets.
Purview Data
Source Administrator -
Can manage all aspects of scanning data into Azure Purview but does not have
read or write access to content beyond those related to scanning. The role does
not have access to the Purview Portal (the user needs to also be in the Data
Reader or Data Curator roles)
Access to
Purview is through Purview Studio. One the home page there are 2 menus. The
quick access to menus in the centre of the page take you to
The left hand
menu has 5 icons
Home- to
see a summary of sources
Sources – to
add and manage sources
Glossary – to
add and manage glossary term collections
Insights – the
visual dashboards of the assets
Management – a
place to perform administrative tasks
There 3 areas
Data Map, Data Catalog and Data Insights
Data Map
The data map
shows your sources in collection grouping of your choice. Many data types are
added by default with more coming all the time. Azure Purview is built on top
of Apache Atlas and with its API you can gain extra functionality.
Once a data
source is add with the relate permissions granted on the asset regular scans
can be scheduled. The is network isolation can give the ability to scan
on-premises and Azure data sources behind a vnet using SHIR ensuring E2E
network isolation.
Governing
your scan has these step
Glossary
This
contains a set of terms the business uses. There might be multiple terms in the
business that mean the same thing.
There are some default attributes that exist and can be enhanced by custom attributed. The default attributes are
Classification and labelling
Labelling of data is
important to aid with communication. Consistency is important. Classifications
can describe
Azure Purview provides a set of default classification rules
which are automatically detected during scanning. Purview uses the
same classifications, also known as sensitive information types,
as Microsoft 365. Azure Purview integrates with Microsoft
Information Protection Sensitivity Label. There is automated scanning
and labelling for files Azure Blob storage, Azure Data Lake Storage Gen 1 and
Gen 2. Automatic labelling for database columns for SQL Server, Azure SQL
Database, Azure SQL Database Managed Instance, Azure Synapse, Azure Cosmos DB. The
default classification rules are not editable although it is possible to define
your own custom classification rules using Regex or custom expressions. Classification
looks at the data e.g. select top 100 from customers for data profiling ,
customer pattern matching , expression matching and applying the classification
tags afterwards.
Data
Catalog
This
enables you to search your data , use workflows in the business glossary and
view data linage for sources in the data ingestion pipeline. It connects with
tools such as Power BI, Azure Data Factory and Azure Synapse. It enables
curations and collaboration.
Data Insights
This is a
key pillar in Purview to enable a single pane of glass in the catalog.
The insight
reports that currently exist are
The Shape Your Future with Azure Data and Analytics digital event on 3rd December 2002 shared how a unified, end-to-end platform for data and analytics builds business agility and resilience. There is an imperative for digital transformation to take place. The speed of that transformation will define an organisations future. It is that tech intensity that Satya Nadella keeps mentioning.
The ability to empower people intelligently, virtually optimise operations and transform products is the future he sees. It is this analytical power that will turn an organisation from being reactive into being proactive. This predictive power will transform organizations. Data is your most strategic asset and it should be listed as its own line as a business asset.
It was announced that Azure Synaspe became generally available.
A tool that I wasn't aware of was Azure Synapse link. Azure Synapse Link enables no complex Extract Transform and Load of data to get to the analytics.
Digital transformation is never done. It is lots of little races to improve things. Once the data foundation is built there is freedom which enables flexibility to reimage the process then automate it. Data is going to define the century in the environment and health care resilience. The ability to test hypothesis's in parallel and experiment every day transforms the efficiency and effectiveness of the world.
The many speakers talked about the core capabilities of Analytics and AI and Data Governance. Microsoft announced Azure Purview, a unified data governance platform. It will change the way businesses grow and enable predictive power. Data governance is key to gaining control over the data. Reimagine data governance as the heart of every successful data project. Data governance reimagined with a business glossary, a trusted data dictionary and no more building manual ontologies with automated classification. As well as a linage function you can look downstream to see where all the data is being used. It see this as being amazingly helpful in supporting business. It is a data map across your data estate that will register, scan and auto classify data. A business glossary helps drive meaning. Azure Purview changes the game in governance and was a missing piece.
A picture tells a 1000 words. Just a bit of fun collating some key points about Data Catalogues.
What is a data catalog? Gartner define this as
“A data catalog maintains an inventory of data assets
through the discovery, description, and organization of datasets. The catalog
provides context to enable data analysts, data scientists, data stewards, and
other data consumers to find and understand a relevant dataset for the purpose
of extracting business value.”
Data Catalogs are
the New Black in Data Management and Analytics (Gartner, 2018)
A data catalog is important to have to record those critical assets that bring value to data. It becomes a library full of core information about your data sources. It can contain a data dictionary and can provide basic statistics about the data. This is a really useful feature being able to explore the data.
Azure Data Catalog documentation is here and the service here
