Azure Purview becomes Microsoft Purview

 A major change was announce yesterday with the future of compliance and data governance combined. The naming has changed from Azure Purview to Microsoft Purview. Watch the announcement here. 

The announcement can be read here . Microsoft talk about the lines between risk roles are blurring. 

• The pandemic 
• Nation-state attacks  
• Remote work 
• Evolving regulations 
• Data sharing 
• Growing CDO responsibilities 
• Governance and compliance

It is interesting that Microsoft mention that the CDO role may go beyond data management and protection to include business intelligence, AI and machine learning. I am not sure how that would work with the chief analytics officer (CAO) and CISO and overlapping roles.

The new Microsoft Purview is explained in the article as it:

• Helps you gain visibility into assets across your entire data estate.
• Enables easy access to all your data, security, and risk solutions. 
• Helps safeguard and manage sensitive data across clouds, apps, and endpoints.
• Manages end-to-end data risks and regulatory compliance.
• Empowers your organization to govern, protect, and manage data in new, comprehensive ways.

The article also talk about enhancing data governance with Microsoft Priva.

Read more about Azure Purview is now Microsoft Purview where it is shared the Microsoft Purview is a comprehensive set of solutions to help you govern, protect, and manage your entire data estate. They have brought together  Azure Purview and Microsoft 365 Compliance portfolio under one brand.

Watch Go Beyond with Microsoft Purview

 

Panel discussion about the CDO seat at the cloud table

There was an excellent EDM Webinar with Microsoft discussing the CDO seat at the cloud table with panellist John Bottega (president of the EDM council) , Karthik Ravindran (General Manager, Enterprise Data) and Mike Flasko (General Manager Data governance and privacy platform) on 29 March.

The session discussed some really important topics around, data governance and data management. Most businesses are on a data modernization journey and often the progress with in business is in small steps. This has the advantage to reduce the risk to business and to help move things forward. There are two  important principles to be considered: 

• Modernization not migration. It is that digital transformation focusing on the data estate to accomplish those outcomes. Incrementally modernizing those data estates into an architecture which enables us to consistently manage and govern our data assets, as well as responsibly democratize those data assets 
• There is someone in the business to champion data, solving problems with data across the enterprise and driving new tools to leverage maximising business value. It is important to unlock the potential of data and make it more accessible to users to be able to accomplish those opportunities. Also thinking about how data is proliferating throughout the business and the need to manage and govern it across the enterprise. 

There has been an organic evolution of the data estates, from the paper driven standards defining data management to technology driven outcomes. The implementations of standards is hugely varied and diverse across the organization. Creating benchmarks for core themes accountable to managing data hasn't been consistent or scalable. There has been a focus on making data management, the operations of data management and data governance, consistent and scalable through intelligent automation. When you start democraticing data it is enabling teams across the company to discover the data,  access it and use the data. Knowing that there are guardrails in place to ensure that the foundations of data management and data governance are being done well at scale,  gives confidence in the data. 

There is a question about how does one manage democratization and acceptable data with security and privacy in place. A couple of key characteristics stood out

• the pure volume of data that we were collecting was not going to scale to the ways we approach things originally
• that transformational power of the cloud to governance to data management

Looking at the problem questions they asked

• why can't inventory and classification, be a one click to click problem that scales up and down with your data estate. 
• why can't classification be intelligent with machine learning infused across that journey for a broad and deep understanding of the data assets that are being collected.

This led to core components the should be looked at

• who is the owner 
• who is accountable is useful for managing a data privacy program and trying to facilitate the data subject rights requests or understand where personal data is held about a particular data subject.
• It is important to have a common data mapping to aid understanding so that can be used to drive everything from privacy program to security posture risk analysis.

Most importantly it is to think about data governance differently.  Data governance has been around for a long time. It is often looked at as being a gatekeeper function versus and enabling function. Many companies don't use the word governance as they feel it frightens people. Mike disagreed with that statement where you think of data governance as an impediment to progress. Governance is critical to progress.

A question was asked about how did you sell the governance concept successfully within Microsoft.

There are many opportunities in this space, especially with the advances that are happening in tech and platform with scaling. To achieve this data management excellence from the beginning it is important to standardize, making consistent data, have scalability, be seamless and simple to use. The core things

• be able to discover data assets in a single catalogue
• standardizing data quality management by having consistent way to measure the quality of the data estate across various data quality dimensions, like completeness and accuracy. Be able to see the health of the data estate across those quality dimensions
• privacy, security and compliance. Compliance is especially important for enterprise standards that an organization has to be compliant with but also with the growing and ever evolving set of regulatory standards that are appearing in the industry.

With all of this in mind companies across many industries have come together to create a playbook of best practice for managing data in the cloud. This is the CDMC.  It was built by the community sharing ideas for best practice. 

How to get started with Data Governance

This is a question that is being asked by many at the moment. The term data governance still comes with the old stigma of control and costly and requiring teams of people. There are those that still believe that going in under the radar and growing once value is shown is a way to start. Then there are those that think we should utilise tools to help with data governance to enable speed of adoption. Myself I think considering business value must be the place to start to have great data and business buy in. After all why do we want to collect the data in the first place. So don't talk about governance but about understanding that data, data erudition, and knowing your current data quality state.  Once the business is in place, embed the process for each use case.

Data governance needs to be simple clear messaging to the business based on an agile method. The benefits of each use case must be measurable and bring value. To be successful it relies on cross team communication and diversity in views. That strong cross functional collaboration, ethical understanding which is sustainable. Data governance underpins all data tasks.

Start with an agenda item in an existing meeting and the program will grow over time. 

Don't follow a framework as every business is different but pull from DAMA (body of knowledge) and CDMC (playbook of best practice )

Start with a data readiness assessment with four areas that looks at the current usage, data quality, how data is managed and business readiness.

Think about the two core questions that are trying to be solved are 

1.             what business problem are you trying to solve

2.             What tooling would help solve the business problem

Start small and create an iterative roadmap based on business value

Reading 

A Guide to Data Governance – Microsoft   https://bit.ly/3hDe5Sn

Chief Data Officer (CDO) Seat at the Cloud Table    https://bit.ly/375px7e

Get started with Azure Purview   aka.ms/azurepurview/getstarted

Microsoft Learn: introduction to Azure Purview   aka.ms/intro-to-azure-purview

Azure Purview Documentation   https://docs.microsoft.com/en-us/azure/purview/

Azure Purview Readiness   Checklist  https://bit.ly/3HIytMy

Self-Service Data Access Policy

Self-service data access policy now in public preview. This enables policies that are auto-generated that control access to underlying datasets.

The business value can be gained by helping with clarity on the consumer requesting access to the data sets. A business is often left asking

1. Which dataset is the user requesting access to?
2. Who must approve the data access request?
3. How many approvers and how long will it take to gain access?
4. Who must provision the data access?
5. is data access transparent?
6. how do I get a policy generated?

The self-service data access request triggers an approval workflow. Thus data discovery becomes easier for a data consumer with Purview on discovery and classification, self service access control. Data stewards and owners keep this information up to date. 

Azure Purview Workflows now in Public Preview

Azure Purview Workflows moved to Public Preview 10 March.  This functionality enables customers to orchestrate the create, update and delete operations of data entities, have validation, and approval of these data entities using repeatable business processes.

The benefit of using Purview workflows is higher quality data, policy compliance, user collaboration, and change tracking awareness across the organization.

As part of public preview, two other features have been enabled

• Self-service data access workflow for hybrid data estate  
• Approval workflow for business glossary

There are two kinds of workflows at the moment:

• Data governance for data policy, access governance, and loss prevention. Scoped at the collection level.
• Data catalog to manage approvals for CUD (create, update, delete) operations for glossary terms. Scoped at the glossary level.

There are workflow templates  to help create those workflows to save time in creating one.

Business terms within an organisation become a critical part of the everyday organisation. To improve efficiency in managing approvals this has been automated. The terms appearing in a business glossary can use the workflow templates to allow approved terms to be added, updated or deleted in the catalog.

The four templates available for the business glossary are:

• Create glossary terms
• Update glossary terms
• Delete glossary terms
• Import terms

Creating a data access request workflow is similar to above . Creating this enables a data consumer to go to the tool and request access. The workflow is followed to enable management of self service data access. The business benefits being speed, automatic approvals and efficiency of those data assets in Purview. 

 Reading

• overview of Azure Purview’s workflow capabilities here. 
• how to manage self-service data access workflows.
• how to manage approval workflow for business terms.
• how to manage requests and approvals generated by workflows.

SQLBits and the case for data governance

SQLBits is fast approaching and in 3 weeks time we have the opportunity to meet the data community in person again. It will be nice to catch up, but also it will be different than before with the conference being partially virtual. It is great to see the opportunity to attend virtually as well as in person. As well as the Covid issue, this makes the event more inclusive.

Data governance is one of the things I am keen to help drive forward because understanding your data better means increased business value and enables innovation. Data governance however is a word that the industry says should not be named. It has connotations of expense and little value for businesses. It is in my mind data erudition, that learning, showing great knowledge about the data and having good quality data. Then using agile techniques to tackle small chunks and iterate through the highest priority business cases  first. It is not about trying to bring all the historic data into a managed state and drilling into full data dictionaries.

With that in mind I am excited about my session with Erwin and Wolfgang with Richard moderating where we will answer questions about Data Governance with Azure Purview. There are two ways to tackle the problem of data governance, through understanding the business problem and through using a tool to help solve part of the problem technologically. Experts in the industry have differing opinions on which to start with first.

Ask your questions for our session via this form https://forms.office.com/r/dTP38LnmsJ

We look forward to seeing you on Friday 11 March from 14:10-15:00. You will find us in room 04.

Data Governance with Azure Purview

SQLBits 8-12 March 2022 is approaching soon. I am really excited to be a part of a session with Erwin de Kreuk and Wolfgang Strasser. The session: Data Governance with Azure Purview - Ask the Experts

To submit your questions in advance for the session we have a Microsoft form to complete.

Azure Purview Dataset provisioning by data owner for Azure Storage (preview)

 

Azure Purview Dataset provisioning by data owner for Azure Storage (preview)

Guide: how a data owner can enable access to data stored in Azure Storage from Azure Purview

To enable access policy enforcement for the Azure Storage account the following PowerShell command needs executing in the subscription where the Azure Storage account resides. It relates to all Azure Storage accounts in that subscription.

# Install the Az module

Install-Module -Name Az -Scope CurrentUser -Repository PSGallery -Force

# Login into the subscription

Connect-AzAccount -Subscription <SubscriptionID>

# Register the feature

Register-AzProviderFeature -FeatureName AllowPurviewPolicyEnforcement -ProviderNamespace Microsoft.Storage

Note: Only new Storage accounts, created in the subscription after the feature AllowPurviewPolicyEnforcement is registered, will comply with access policies published from Purview.

More details to read about this option

The Chief Data Officer seat at the table

Microsoft have written a white paper called Microsoft Azure: The Chief Data Officer (CDO) Seat at the Cloud Table It is an interesting paper to read.

The Whitepaper looks at four areas

1. Journey
2. Framework
3. Product
4. Resources

The document mentions other related capabilities that need to be included in a data governance program:

• Data Discoverability 
• Data Quality Management
• Data Access Management 
• Data Compliance 
• Data Lifecycle Management 
• Data Health Scorecards  

Microsoft’s data governance approach is listed as

• Set the scope of data governance for your organization
• Set enterprise data governance requirements through policies and standards
• Set ownership and accountability for data governance
• Start with a unified, metadata-driven vision with automation
• Iterate, not big bang
• Educate and enable change
• Monitor and revise

The Data Management Capabilities Model Framework (DCAM) core capabilities closely align the Microsoft Framework.  DCAM establishes the data strategy, position the business case, implement the operating model, ensure funding and supportive organizational collaboration.

The holistic list of capabilities highlighted in the CDMC from the EDM Council is:

• Data Cataloguing and Discovery 
• Data Classification 
• Data Ownership 
• Data Security 
• Data Sovereignty and Cross-Border Data Sharing 
• Data Quality
• Data Lifecycle Management 
• Data Entitlements and Access Tracking  
• Data Lineage  
• Data Privacy 
• Trusted Source Management and Data Contracts 
• Ethical Use and Purpose 
• Master Data Management

 

PASS Summit Key note Unified Data Governance with Azure Purview

Raghu Ramakrishnan CTO Data, Technical Fellow, Microsoft spoke at PASS Community Summit in November and explained the next part of the vision, policy, for data governance. Microsoft are seeing data governance as the emerging data pillar. Operational databases, unified analytics platform, and unified automated data governance. The unified part is the important element going forward, a unified single pane to extend governance across the entire data estate. Automated data classification to remove the PII headache of missing personal data and pushing the control up the stack to knowledge workers. Microsoft intend to have dynamic data providence that is fully integrated with the 6 responsible AI principles.  Azure Purview will operate a Central RBAC control and is the governing permission future state for SQL Server with full propagation. With AI integrated the policy feature will be human readable. The link to watch the session .

Data governance is increasingly interdisciplinary and the discovery of data core to a business. Questions often asked: what data do we have? where did the data originate? can I trust the data?

Compliance is an area which had been a major area of data governance. Questions often aske here:

what’s my exposure to risk? is my data usage compliant? how do I control access and use the data? what is required by regulation X?

Raghu talked about the data governance journey through the lens of GDPR Compliance. There approach was to create a 'Data Map' of all data across Microsoft and use that map to support GDPR compliance. The data discovery looking at search and discover, information supply chain, steward/curators and business glossary. Then looking at the data use governance and policy author/manage, reporting, access and governance enforcement and industry compliance. These two areas were built on intelligent data inventory - built on a data map with automated structure & lineage collection, automated & custom classification and publication / subscription APIs. 

Purview data catalog is a self service tool filled with details from knowledge workers. Areas include:

• self-service search and browse
• curated and standardized business glossaries
• interactive lineage visualization
• simplified data curation and stewardship

The data estate insights currently show these

• data asset distribution
• business glossary
• data classification and labelling
• data location and movement (in progress)

The Microsoft vision is: data in the Microsoft cloud is always governed and beyond Azure, Purview offers a single pane to extend governance across the entire data estate.

Still looking through the lens of GDPR Compliance data classification is an important feature

Dynamic lineage deep dive

He talked about the increased efficiently of extracting of dynamic SQL provenance and the 6 responsible AI areas of fairness, inclusiveness, reliability and safety, transparency, privacy and security and accountability. Talking about responsible AI and provenance with  machine learning  (ML) training and audit with the provenance of ML models as a requirement. There are a number of challenges address to enable this. 

Centralized data access control

Proactive governance controls look at things like

Policy enforcement inside data services - access control was explained (in the early stages so may change)

In the future there was mention of an ABAC Policy language ABAC = RBAC + Conditions. A human readable policy language for business users like data officers or data owners. A policy statement can be represented as a tuple of {Effect, Action, Data resource, Subject, Condition}.  The propagation of Purview polices to data repositories is asynchronous in design with Purview as the single source of the truth. SQL pull updates asynchronously,  and updates are thus not immediately visible locally like AAD logins. 

The summary of the presentation was Purview is creating a new data pillar of unified governance across the entire data state. It is deeply integrated with SQL Server, extending its governance capabilities significantly.

T-SQL Tuesday #144 – Data Governance reimagination - Wrap up

This month’s T-SQL Tuesday attracted some great responses! Thank you to everyone who participated!

My invitation for this month’s #tsql2sday was 3 fold on sharing your experiences on data governance

• The current cost of data governance versus its benefits
• The amazing things data governance has enabled you to achieve or will enable you to achieve in the future
• The potential uses for Azure Purview within your estates and the automated deployment options for that

Rob Farley published a post in reply

http://blogs.lobsterpot.com.au/2021/11/09/being-sure-of-your-data/

Rob raises some key points 

• But the checks that we do are more about things that the database can allow, but are business scenarios that should never happen.
• You need to discover which situations cause people not to trust the data.
• Data quality can lead to the trust, but only when it has been demonstrated repeatedly over time. Trust must be earned

Deborah Melkin published a post in reply

https://debthedba.wordpress.com/2021/11/09/t-sql-tuesday-144-data-governance/

Deborah Melkin talks about the switch to implement data governance.

• It is about understanding your data from both the micro and macro level
• It’s understanding where our data lives (data assets) and how data flows through data sources (data lineage) as well as how it’s consumed and used (data catalogs and data profiling). More importantly, this is knowledge that can be shared to make data even more valuable.
• When you start expanding the number of databases and the complexity of how your systems work, the job of governance becomes a lot harder
• Getting started with data governance seems like a very daunting task.

Data Governance is a broad topic with many different areas which can be be seen from the replies. There is plenty for us to get started with and I'm looking forward to using Azure Purview to help with this. 

Thank you for taking the time to post insightful posts. That is the wrap up. If I’ve missed anyone please let me know and I’ll update the post.

Ignite Innovate Anywhere From Multicloud to Edge

Innovate Anywhere From Multi-cloud to Edge with Scott Guthrie shares a raft of technical updates. The themes

• Hybrid and multicloud
• End-to-end data platform
• Cloud native development
• Developer velocity

Hybrid and multicloud updates announced

• Deeper integration with VMware vSphere and Azure Stack HCI
• Azure Virtual Desktop on Azure Stack HCI
• Azure Arc enabled data services updates
• Extension of Microsoft Defender to AWS

A data platform with end to end capability

SQL Server 2022 was announced

Read more: Announcing SQL Server 2022 preview: Azure-enabled with continued performance and security innovation

Azure Synapse Analytics announcements

• Azure Data Explorer
• Synapse Link SQL Server 2022
• Synapse Link Dataverse (GA)

Read More: Ignite 2021 Announcements: Accelerate time-to-insight with Azure Synapse Analytics

Amazing to see Azure Purview built into the new SQL Server platform.

Great to see Azure Purview and data governance mentioned everywhere. 

T-SQL Tuesday #144 – Data Governance reimagination

Data governance is a topic that has raised its head again in the last year, with the introduction of a GA service called Azure Purview.  Data governance is not a new topic in the realms of data management.  I think over the last few years data governance has had a focus on meeting data protection law, government legislation and formalised control and standards. Then with data sovereignty issues being everyday considerations within the global market place, storage locations in the cloud and the likes of the General Data Protection Act, data governance has been very focused on meeting legislative requirements.

There has been a substantive cost involved in setting up data governance within an organisation to avoid those heavy fines if a data breach were to occur. I think because of this many organisations compartmentalise personal data systems.

A change in perspective, a reimagination, of data governance is occurring. Data governance is really about ‘data erudition’, showing an interest in learning about the data we have, improving the quality and creating a more productive and trusted data asset. Starting small and incrementing data change in a way that matches the business need, is a way to gain targeted business value. Azure Purview provides this great opportunity for us to start at the beginning and create a data catalog, data inventory, data dictionary and much more in an automated way.  Trusting the data quality, knowing your risks and what data you have sets your business up for success.

My invitation to you for this month’s #tsql2sday is…

I want to invite you to share your experiences on data governance

• The current cost of data governance versus its benefits
• The amazing things data governance has enabled you to achieve or will enable you to achieve in the future
• The potential uses for Azure Purview within your estates and the automated deployment options for that

*** The Rules ***

• Your post must be published on Tuesday 9th November 2021. This counts as long as it’s still Tuesday anywhere in the world.
• Include the T-SQL Tuesday Logo and make it link to this invitation post.
• Please send me a tweet @victoria_holt with a comment to this post with a link to your own so I know where to find it.
• Please tweet about your post using the #tsql2sday hashtag.

Data Erudition

I have been thinking about data governance, Azure Purview and data research. How these areas link together and enable innovation, that democratisation of data and holistically thinking about data as a strategic asset. A combination of above in my diagram covers my thoughts.

Azure Purview Generally Available

The maximize the value of your data in the cloud: achieve unified data governance with Azure Purview digital event with Rohan Kumar, Corporate Vice President Azure Data and Mike Flasko, General Manager of Azure Data Governance Platform on 26 September 2021 was an exciting event.  It enables the new reimagined agile data governance world to move forward.

The event explored Azure Purview, a unified data governance solution that gives you a holistic, up-to-date map of your entire data estate. The general availability of Azure Purview was announced bringing that improve automated governance to the fore. 

The product launched with an area in preview. 

There is automated data discovery using Purview data scan supporting hybrid data landscapes with classification and lineage. There are 200+ data classifiers with 35 data sources. The data map graph describes the data assets and relationships across the estate with fine gained access controls. The lineage feature is really important as it enables root cause analysis to happen and ensures data lineage is available via visualisation. Being able to search, browse and curate data enables you gain more understanding of your data. 

Business context with data is important and to have a business glossary enables relevant business terms to be connected. There is hierarchy support and an integrated approval workflow.

There are a large set of connectors that enable data scanning currently available and there is pubic preview of more data options for data scanning for Google Cloud, Erwin, Salesforce, IBM DB2 and Cassandra. Data scanning options coming soon are snowflake, SAP HANA, PostgresSQL, MongoDB and MySQL.

Azure Purview is growing in functionality all the time and Purview Data Insights is in preview. This looks at 

• data asset distribution 
• sensitive data
• data scanning coverage 
• business glossary utilisation

The insights capability is useful for CDOs to gain a high level picture of the data estate.

A glimpse of the exciting roadmap to come was also shared.  

The Azure Purview features are

There a lots of resources already available

Azure Purview overview 

Introduction to Azure Purview Microsoft Learn to start your learning journey.

Official Purview blog for updates

Billing for Azure Purview will start on 1st November 2021. Pricing 

Announcements are detailed here

Customer Stories shared

Microsoft Customer Story-Danish pump manufacturer develops sustainable water solutions with unified data governance from Azure Purview

Microsoft Customer Story-Heathrow boosts operational efficiency and improves decision making with Azure Purview

Microsoft Customer Story-illimity optimizes data governance and streamlines compliance with Azure Purview

Maximize the value of your data with Azure Purview

There is a digital event coming on 28 September 2021 9-10am pacific time. The event is about achieving unified data governance with Azure Purview.
Join Microsoft Corporate Vice President Rohan Kumar at this free digital event for demos and deep dives.
Register now to:

• Learn to create a comprehensive, automated map of all your data.

• See how Azure Purview works with Azure Synapse Analytics, Power BI, and the rest of your data estate to deliver timely, reliable insights.

• Watch in-depth demos of product features including Azure Purview Data Map and Data Catalog.

• Ask Azure experts your data governance questions in the live Q&A.

 

Azure Purview August updates

There have been some exciting changes to Azure Purview announced. These changes relate to charging and permissions. 

Elastic data maps

The data map is the foundation for data discovery. The data map has two components the throughput created by CRUD operations and storage of the metadata.

The data map can now grow elastically starting at one capacity unit. A capacity unit includes a throughput of 25 operations/sec and the metadata storage scales in increments of 2 GB. Purview Data Map can automatically scale up and down within limits . This new charging model makes Purview much more user friendly and less costly to set up and run.

Access Control in Azure Purview

Note: only applies for Purview accounts created on or after 18 August 18 2021,

A collection is a tool to group assets, sources, and other artifacts into a hierarchy for discoverability and to manage access control. Collections are used to organise and manage assets.

The are various roles that exist:

• Collection admins - can edit Purview collections, their details, and add sub collections. They can also add users into other Purview roles on collections where they're admins.
• Data source admins - can manage data sources and data scans.
• Data curators - can create, read, modify, and delete catalog data assets and set up relationships between assets.
• Data readers - can access but not modify the data.

There is a great video to watch from W Strasser explaining this new data plane RBAC (role based access control) catalog permission. Being able to fine tune the access to the collections brings with it great advantages. Currently collection names can't be updated or deleted. 

Purview Readiness one-pager checklist

Starting with Azure Purview requires a few prerequisites. The checklist  sets out 4 phases. 

Identify where to start to establish a Governance baseline foundation for your organization for general cloud governance. 

Azure Purview Readiness Checklist

Purview-Deployment-Checklist will help prepare for data governance and data democratization in your environment.

There are four sections

• Readiness
• Build foundation
• Register data sources
• Curate and consume data

The Azure Purview deployment prerequisites can help with those proof of concept deployments

Azure Purview automated readiness checklist

A set of scripts have been written to help evaluate your exciting environment for missing configuration that might prevent data sources being scanned. The PowerShell scripts are

• Azure-Purview-automated-readiness-checklist.ps1
• Azure-Purview-automated-readiness-checklist-csv-Input.ps1

Azure Purview - A Data Catalog

Data Catalogs are becoming an essential component in the new data world. They are an inventory of an organisations data assets. The meta data collected helps in finding the most appropriate data at speed, know what data is held, the security levels and lineage of data. Microsoft have tool in preview that helps with data governance  called Azure Purview.

Azure Purview is a unified data governance service that helps you manage and govern your on-premises, multicloud and software-as-a-service (SaaS) data. Easily create a holistic, up-to-date map of your data landscape with automated data discovery, sensitive data classification and end-to-end data lineage. Empower data consumers to find valuable, trustworthy data.

Once installed it moves into the hands of data governors. There are predefined data plane roles dictating who can access what.

Purview Data Reader - access to the Purview portal and can read all content except for scan bindings

Purview Data Curator - access to the Purview portal and can read all content except for scan bindings, can edit information about assets, classification definitions and glossary terms, and can apply classifications and glossary terms to assets.

Purview Data Source Administrator - Can manage all aspects of scanning data into Azure Purview but does not have read or write access to content beyond those related to scanning. The role does not have access to the Purview Portal (the user needs to also be in the Data Reader or Data Curator roles)

Access to Purview is through Purview Studio. One the home page there are 2 menus. The quick access to menus in the centre of the page take you to

• Knowledge centre
• Register sources
• Browse assets
• Manage glossary

The left hand menu has 5 icons

Home-  to see a summary of sources

Sources – to add and manage sources

Glossary – to add and manage glossary term collections

Insights – the visual dashboards of the assets

Management – a place to perform administrative tasks

 

 

There 3 areas Data Map, Data Catalog and Data Insights

Data Map

The data map shows your sources in collection grouping of your choice. Many data types are added by default with more coming all the time. Azure Purview is built on top of Apache Atlas and with its API you can gain extra functionality.

Once a data source is add with the relate permissions granted on the asset regular scans can be scheduled. The is network isolation can give the ability to scan on-premises and Azure data sources behind a vnet using SHIR ensuring E2E network isolation.

Governing your scan has these step

• Register your source
• Apply and set up your credentials
• Set up and run your scan
• Discover your SQL server data

Glossary

This contains a set of terms the business uses. There might be multiple terms in the business that mean the same thing.

• synonyms - different terms with the same definition
• related - different name with similar definition

There are some default attributes that exist and can be enhanced by custom attributed. The default attributes are

• Name
• Definition
• Data stewards
• Data experts
• Acronym
• Synonyms
• Related terms
• Resources

Classification and labelling

Labelling of data is important to aid with communication. Consistency is important. Classifications can describe

• A type of data that exists in a data asset or schema to help identify the content of a data asset.
• It could describe a data preparation process
• Can also help with compliance

Azure Purview provides a set of default classification rules which are automatically detected during scanning.  Purview uses the same classifications, also known as sensitive information types, as Microsoft 365. Azure Purview integrates with Microsoft Information Protection Sensitivity Label. There is automated scanning and labelling for files Azure Blob storage, Azure Data Lake Storage Gen 1 and Gen 2. Automatic labelling for database columns for SQL Server, Azure SQL Database, Azure SQL Database Managed Instance, Azure Synapse, Azure Cosmos DB.  The default classification rules are not editable although it is possible to define your own custom classification rules using Regex or custom expressions. Classification looks at the data e.g. select top 100 from customers for data profiling ,  customer pattern matching , expression matching and applying the classification tags afterwards.

 

 

Data Catalog

This enables you to search your data , use workflows in the business glossary and view data linage for sources in the data ingestion pipeline. It connects with tools such as Power BI, Azure Data Factory and Azure Synapse. It enables curations and collaboration.

Data Insights

This is a key pillar in Purview to enable a single pane of glass in the catalog.

The insight reports that currently exist are

• Asset Insights
• Scan Insights
• Glossary Insights
• Classification Insights
• Sensitivity labelling insights
• File Extension Insights