Modern Analytics and Governance at Scale

Microsoft have released a white paper talking about Analytics and Governance (MAG) at Scale to learn how to successfully build an end-to-end analytics ecosystem aligned to datamesh, datafabric, and datahub. Starting with an enterprise data strategy in place, this enables your organization vision and roadmap to accelerate value creation drawing on your data. This lays the groundwork for building the three fundamental pillars of Modern Analytics and Governance (MAG) at Scale: 

1. Enterprise Data Governance: This includes lineage, classification and policy. Then data ordering and moving to rapid access to data
2. Data Management Foundation: Lakehouse as a foundation includes automated data management, ingestion, standardization, quality, metadata registration and access provisioning
3. Domains and Data Products: Autonomous domains provisioned, analytic federations , ingestion and curation for analytics patterns

This is a transformative approach to the modern analytics ecosystem. Capabilities enabled at scale are

Including data governance in solutions has these objectives

• Discover and describe data
• Automatically secure data
• Democratise data access
• Provide data leaders an understanding of their data

To move towards that enterprise data governance state it is important to build a data foundation . The whitepaper mentions four ways to start with this

1. Creating metadata
2. Mapping data assets
3. Cataloging data assets
4. Automating governance and security

The paper goes on to share an implementation plan and much more for Microsoft Purview . 

MVP Summit 2023

It is MVP Summit this week. 

It is week to catch up with friends and learn. I hope everyone at MVP Summit has fun, learns a lot, provides useful feedback and is understanding. Thanks to the team for organising the event. Sad I am not there but I see my name on the board. #MVPBuzz

Data Toboggan - Cool Runnings 2023

Submit a session for Data Toboggan – Cool Runnings, our 3rd summer edition on  Saturday 24 June 2023. 

There are two session lengths for submissions. These are: 

• 45 mins
• 10 mins

The Call for Speakers is open and awaiting your submissions on Azure Synapse related content. If you are unsure what to submit check out our YouTube channel full of past event sessions. We would love to see your sessions, new speakers or experienced speakers, so submit now.

CfS: https://sessionize.com/data-toboggan-cool-runnings-2023/

You Tube: https://www.youtube.com/c/DataToboggan

Web: https://www.datatoboggan.co.uk/

In a Nutshell Concept Map

There is lots of talk about DAMA-DMBOK2 and DCAM and this can be confusing for people who are new to data governance and data management. There are many definitions describing what data governance is. For example Wikipedia states

'Data governance is a term used on both a macro and a micro level. The former is a political concept and forms part of international relations and Internet governance; the latter is a data management concept and forms part of corporate data governance.'

I came across this article  Data Management and Data Governance in a Nutshell which has a useful concept map of data management and data governance definitions. DAMA and  DCAM are fairly well aligned now.

Microsoft achieves first CDMC certification

Microsoft achieves first Cloud Data Management Capabilities (CDMC) certification for Microsoft Purview. This has the capabilities to accelerate business move to the cloud.

The 14 Key Controls and Automations are a part of the EDM Council’s Cloud Data Management Capabilities framework formulated as a best practice to help all industries accelerate the migration of sensitive and non-sensitive data to the cloud with confidence.

There are 14 controls that make up the framework for automations

Governance and accountability

1. Data Control Compliance must be monitored for all data assets containing sensitive data through metrics and automated notifications.

2. The Ownership field in a data catalog must be populated for all sensitive data or otherwise reported to a defined workflow.

3. A register of Authoritative Data Sources and Provisioning Points must be populated for all data assets containing sensitive data.

4. The Data Sovereignty and Cross-Border Movement of sensitive data must be recorded, auditable, and controlled according to defined policy.

Cataloging and classification

5. Cataloging must be automated for all data at the point of creation or ingestion, with consistency across all environments.

6. Classification must be automated for all data at the point of creation or ingestion and must always be on.

Accessibility and usage

7. Entitlements and Access for Sensitive Data must default to the creator and owner and access must be tracked for all sensitive data.

8. Data Consumption Purpose must be provided for all Data Sharing Agreements involving sensitive data.

Protection and privacy

9. Appropriate Security Controls must be enabled for sensitive data and evidence must be recorded.

10. Data Privacy Impact Assessments must be automatically triggered for all personal data according to its jurisdiction.

Data lifecycle

11. Data Quality Measurement must be enabled for sensitive data with metrics distributed when available.

12. Data Retention, Archiving, and Purging must be managed according to a defined retention schedule.

Data and technical architecture

13. Data Lineage information must be available for all sensitive data.

14. Cost Metrics directly associated with data use, storage, and movement must be available in the catalog.

SQLBits 2023

SQLBits 2023 at the ICC in Wales, was an event with so many moving parts.  It was the biggest ever with 2576 registrations. There were people there from many countries United States, Denmark, Norway, Switzerland, Germany, Sweden, France, Netherlands, Austria, Spain and Poland as well as the UK. It was both In-Person and Virtual.  We have Community Corner, the BitsBuddies, helpers, a quiet zone, prayer room, a counsellor on site and more. I have attended every SQLBits and helped for 10 of those. 

So many things are amazing about #SQLBits. For me it is the sharing of ideas, community and helping new people join the community. I think we often forget how hard it was when we first started attending events, standing around feeling disconnected and no one to talk to. Being female at events then was very hard. It was amazing to be a helper again and also one of the #Bitsbuddies #community .  See you next year.

SQLBits Business benefits of good governance

I am am pleased to be talking about the business benefits of good governance at SQLBits. My session includes discussion on the CODEX, a framework to help you think holistically about the management of data and database systems in an agile approach.