Welcome

Passionately curious about Data, Databases and Systems Complexity. Data is ubiquitous, the database universe is dichotomous (structured and unstructured), expanding and complex. Find my Database Research at SQLToolkit.co.uk . Microsoft Data Platform MVP

"The important thing is not to stop questioning. Curiosity has its own reason for existing" Einstein



Friday, 24 June 2022

Cool Runnings 2022 Piste Maps

The wait is over the Cool Runnings 2022 Piste Maps are available. Not one but 2 piste maps, one for the Synapse room and one for the Analytics room . We have more then one slope that enables predictive analytics as we are running 2 tracks.

Register: https://bit.ly/DT-CR22Register Agenda: https://bit.ly/DTCR2022Agenda #azuresynapse @Azure_Synapse




Thursday, 23 June 2022

An interactive word cloud full of sessions

For our Data Toboggan Cool Runnings 9 July event we continue to evolve our analytics.

Did you interactively want to see what is included in the sessions. Check out the bottom of agenda page.

Agenda: https://bit.ly/DTCR2022Agenda



Don't forget to Register at the same time: https://bit.ly/DT-CR22Register


Monday, 20 June 2022

Session Content for Data Toboggan Cool Runnings 2022

There is some amazing content to be shared in our next conference so why not spend the day learning  with 2 tracks of content to choose from. 









Register: https://bit.ly/DT-CR22Register

Agenda: https://bit.ly/DTCR2022Agenda 





Saturday, 18 June 2022

Microsoft Purview Data Estate Insights

Microsoft Purview Data Estate Insights will be generally available 1st July 2022. This is an important feature to help the Chief Data Officer (CDO) gain insight about the data estate. The data estate insights helps an organisations connect to diverse hybrid data sources. There will be a new dashboard for Data stewardship, where you can see key metrics about your data estate's health and performance.


The Data Estate Insights covers key insights:

Health – key metrics that pose governance risks or risks to their catalog’s adoption with a data stewardship report

Inventory and Ownership - detailed drilldowns of asset distribution by department and ability to remediate in-app, with the Assets report.

Curation and Governance – details of classified, labelled, or termed assets and where tags may be missing, indicating a potential governance gap, starting with Classification, Glossary and Label reports





 








Actionable insights with KPIs shows what percentage of the data estate does not have assigned data owners and  how many assets are fully, partially, and not “curated”. This can help indicate if there is a significant return on investment. A scorecard gives an idea of the criticality of risks on the data estate.















These metrics can be drilled into further in a tree map. There is also the availability of trend analysis of key governance metrics. You can export the data for offline tracking and follow ups.

Note: Insights generation is enabled at provisioning. 

Thursday, 16 June 2022

Tuesday, 14 June 2022

Thursday, 2 June 2022

Tech in Ten at SQLBits

Great to see SQLBits post this. 

We keep getting more and more from the team at @venturi_ltd  with their Tech in 10 Podcast - Recorded at SQLBits!๐ŸŽง

from discussed and more #data-related subjects, as well as SQLBITS!๐Ÿงก Listen on any podcast platform๐ŸŽ™️ https://pod.co/tech-in-ten/ep-9-dr-victoria-holt-enterprise-data-architect-coeo



https://pod.co/tech-in-ten/ep-9-dr-victoria-holt-enterprise-data-architect-coeo

Wednesday, 1 June 2022

A summary of Microsoft Purview tools


The new Microsoft Purview is a comprehensive set of solutions to help govern, protect, and manage your entire data estate. This new brand family combines the capabilities of the former Azure Purview and the Microsoft 365 Compliance portfolio, providing unified data governance and risk management. The benefits are it
  • helps gain visibility into assets across the entire data estate.
  • enables easy access to all data, security, and risk solutions. 
  • helps safeguard and manage sensitive data across clouds, apps, and endpoints.
  • manages end-to-end data risks and regulatory compliance.
  • empowers organization to govern, protect, and manage data in new, comprehensive ways.
The full suite of combined products in the Microsoft Purview Suite announced recently here

There are 2 areas: 

Risk & Compliance – for risk, compliance and legal teams
Unified data governance – for data consumers, data engineers, data officers

The various tools are summarised below.

Microsoft Purview Audit

Helps respond to security events, forensic investigations, internal investigations, and compliance obligations. Microsoft Purview provides two auditing solutions: Audit (Standard) and Audit (Premium).





A comparison of the key capabilities

Communication compliance helps with complying with internal policies and regulatory compliance requirements. It has focused policy templates and a flexible workflow, with actionable insights to quickly resolve detected compliance issues.

 

Microsoft Purview Compliance Manager

This is a feature in the Microsoft Purview compliance portal that helps with management of organizational compliance requirements. Compliance Manager covers the entire journey, from taking inventory of the data protection risks to managing the complexities of implementing controls, staying current with regulations and certifications, and reporting to auditors.



Microsoft Purview Customer Lockbox

Customer Lockbox brings you into the approval workflow process that Microsoft uses to ensure only authorized requests allow access to your content. The audit record is logged in the audit log when a person in your organization approves or denies a Customer Lockbox request.Microsoft Purview Data Catalog

Data Catalog is an enterprise-wide metadata catalogue that enables data asset discovery. It’s a fully-managed service that lets you register, enrich, discover, understand, and consume data sources.




Microsoft Purview Data Connectors

Supported data sources , file types, and scanning concept



Microsoft Purview Data Lifecycle Management

Manage information lifecycle and records intelligently through

  • Data retention and deletion: Manage risk and liability by keeping only what is needed across the entire digital estate.
  • Records management: Manage high-value records while meeting the legal and regulatory requirements.
  •  Automated policies: Classify and govern data at scale based on file properties, the presence of sensitive data, or machine learning.
  • Proof of compliance: Demonstrate compliance with label analytics insights, defensible disposal, and rich audit trails.


Microsoft Purview Records Management

To manage legal obligations to demonstrate compliance with regulations, increase efficiency with regular disposition of items that are no longer required to be retained, no longer of value, or no longer required for business purposes. The capabilities for Microsoft 365 services and apps:

  • Label content as a record
  • Migrate and manage your retention requirements with file plan
  • Configure retention and deletion settings with retention labels
  • Start different retention periods when an event occurs 
  •  Review and validate disposition
  • Export information about all disposed items
  • Set specific permissions

Microsoft Purview Data Map

The Microsoft Purview data map provides the foundation for data discovery and data governance. It captures metadata about data present in analytics, software-as-a-service (SaaS), and operation systems in hybrid, on-premises, and multi-cloud environments. The Microsoft Purview data map stays up to date with its built-in scanning and classification system.

Microsoft Purview Double Key Encryption

Double Key Encryption (DKE) uses two keys together to access protected content. Microsoft stores one key in Microsoft Azure, and you store the other key.  

Microsoft Purview eDiscovery (Standard and Premium)

Electronic discovery, or eDiscovery, is the process of identifying and delivering electronic information that can be used as evidence in legal cases. Microsoft Purview provides three eDiscovery solutions: Content search, eDiscovery (Standard), and eDiscovery (Premium).







Microsoft Purview Information Barriers

A compliance solution that allows two-way communication to be restricted and collaboration between groups and users in Microsoft Teams, SharePoint Online, and OneDrive for Business.

Microsoft Purview Information Protection

To enable discovery, classification, and protect sensitive information wherever it lives or travels.These information protection capabilities tools are to know your dataprotect your data, and prevent data loss.



Microsoft Purview Data Loss Prevention

Identify and help prevent risky or inappropriate sharing, transfer, or use of sensitive data on endpoints, apps, and services. Microsoft Purview Insider Risk Management

A compliance solution that enables detection, investigation, and act on malicious and inadvertent activities in your organization. Insider risk policies allow the types of risks to be defined to identify and detect, including acting on cases and escalating cases to Microsoft eDiscovery (Premium) if needed. 



Microsoft Purview governance portal

Microsoft Purview is a unified data governance service that enables a foundational understanding of on-premises, multi-cloud, operational and SaaS data.

Microsoft Purview compliance portal

The Microsoft Purview compliance portal provides easy access to the data and tools you need to manage to your organization's compliance needs.


In Microsoft Purview Data Estate Insights, you can get an overview of the assets that have been scanned into the Data Map and view key gaps that can be closed by governance stakeholders, for better governance of the data estate.

Microsoft Purview Customer Key  Customer Key only encrypts data at rest in the cloud. It provides extra protection against viewing of data by unauthorized systems or personnel, and complements. It helps meet regulatory or compliance obligations for controlling root keys. 






Microsoft Priva is a new category of features in Microsoft Compliance Centre, providing solutions for data privacy compliance

Priva Subject Rights Requests

Subject Rights Requests, sometimes called data subject access requests (DSARs), or consumer rights requests,  enquires are sorted with workflow, automation, and collaboration capabilities for helping data search for subject data, reviewing findings, appropriate file collection, and reports. 

Priva Privacy Risk Management

Privacy Risk Management enables the set up of policies that identify privacy risks in the Microsoft 365 environment and enable easy remediation. Privacy Risk Management policies are meant to be internal guides to 

  • Detect overexposed personal data so that users can secure it.
  • Spot and limit transfers of personal data across departments or regional borders
  • Help users identify and reduce the amount of unused personal data that you store.


Workflows - preview feature

Workflows are automated, repeatable business processes that users can create within Microsoft Purview to validate and orchestrate CUD (create, update, delete) operations on their data entities. Enabling these processes allow organizations to track changes, enforce policy compliance, and ensure quality data across their data landscape.

Policies - preview feature

Access policies in Microsoft Purview enable management access to different data systems across the entire data estate. Data access policies can be enforced through Purview on data systems that have been registered for policy. 

Thursday, 26 May 2022

Reading UK Regional Spotlight event for Microsoft Build

Today, 25 May 2022, I was at the Microsoft Build regional spotlight event at Reading in the UK.

I had the amazing honour to speak at the event. I was a part of  two very different sessions. 

Where are all the cloud and data engineers? with Richard Conway and Dr Victoria Holt

Photo credit: Craig Porteous

The number of cloud and data engineers currently on the market are too few to satisfy the demand for the global digital transformation agenda. Where are they all? How do we entice new people into this exciting field and better prepare new graduate engineers to use cloud patterns and practices?

In the table topic we talked about the shortage of cloud and data engineers. The possible reasons for this and discussed actionable insights on how business and people can help increase the number engineers and how it is a continuous learning journey.  We answered questions poised by the in person and online attendees. The session was recorded and will be posted for on demand viewing here

Gain agility with an integrated data platform with Pragati Jain, Dr Victoria Holt, Richard Conway and Andy Cross








Photo credit: Andy Cutler


How you can accelerate innovation on your application with an integrated data platform that adapts to your needs with Hybrid and multi-cloud capabilities?

We started this ask the expert session discussing the Microsoft Intelligent Data Platform that was announced yesterday and told a story of the apps from the core foundations of data governance, Microsoft Purview, moving through to Azure Synapse to digital twins and Power BI. We answered questions from the in person audience and online attendees on various topics.

The session was recorded and will be posted for on demand viewing here

It was a very rewarding day with plenty of opportunity for learning and making new friends. I would like to thank  the other speakers in my sessions for their amazing help,  the amazing tech team, producers, other MVPs , and the audience for selecting our sessions to attend.

Tuesday, 24 May 2022

Microsoft Build Keynote 2022

 The Microsoft Build keynote announced a raft of new features. Satya took a whistle top of new features within 10 areas.








So many new features but these are the highlights of features i am looking forward to using

Microsoft Intelligent Data Platform is now generally available. This new, integrated platform unifies operational databases, analytics and data governance.  Read about it hear.


The platform accelerates innovation, achieves agility and is built on a trusted platform.

Microsoft Purview, the unified data governance platform Satya said

  • Creates a unified map of your data across hybrid sources
  • enables discovery of data that powers business insights
  • enables insight into sensitive data to be gained across your organization






Datamart in Power BI is a new self-service capability included with Power BI Premium that enables users to uncover actionable insights through their own data without any help from IT teams. Power BI integrated governance allows users to apply Microsoft Purview Information Protection sensitivity labels, track data lineage and impact, or retrieve all Datamarts and metadata via automation (scanner APIs).

Public Preview of Azure Synapse Link for SQL . Azure Synapse Link for SQL is an automated system for replicating data from your transactional databases (both SQL Server 2022 and Azure SQL Database) into a dedicated SQL pool in Azure Synapse Analytics. 

Microsoft Purview Data Estate Insights will be generally available July 1, 2022. I will share more details about this in another blog and the new dashboards now available.

The public preview of SQL Server 2022 . This includes link feature for Azure SQL Managed Instance, Azure Synapse Link for SQL, integrated Microsoft Purview and more. Read about it here  























Microsoft Graph is empowering many areas and there are always more Microsoft AI and OpenAI features. There are some interesting metaverse developments. So how to build a metaverse for Data Toboggan is the question?




Microsoft Build book of news  https://news.microsoft.com/build-2022-book-of-news/ 






Microsoft Build Regional Spotlight

It is Microsoft Build this week and this year there are regional spotlights in 5 countries France, Germany, Japan, Latin America and UK.  For the UK tech community the agenda is packed full of different types of  sessions. 













I am involved in a Table Topic session with Richard Conway. This is a live interactive session inspired by community experts to learn best practices and discover helpful resources. 

The session is

Where are all the cloud and data engineers?

Wednesday, May 25 | 1:00 PM - 1:30 PM British Summer Time

This session is hosted in a Microsoft Teams Meeting and will be recorded and posted for on demand viewing on the public event website. The number of cloud and data engineers currently on the market are too few to satisfy the demand for the global digital transformation agenda. Where are they all? How do we entice new people into this exciting field and better prepare new graduate engineers to use cloud patterns and practices?

There are also Ask the Expert sessions to enable attendees to connect with experts to directly have questions answered live. I am helping out as a moderator for a couple of sessions as well.

Making Data Visualisation simple with DevOps practices - Speaker: Ian Pike and  Liping Huang

Wednesday, May 25 | 1:45 PM - 2:15 PM British Summer Time

Gain agility with an integrated data platform - Main Speaker: Pragati Jean, Dr Victoria Holt, Richard Conway and Andy Cross

Wednesday, May 25 | 3:00 PM - 3:30 PM British Summer Time

This session is hosted in a Microsoft Teams Meeting and will be recorded and posted for on demand viewing on the public event website. In this Q&A session, you can learn from our experts on how you can accelerate innovation on your application with an integrated data platform that adapts to your needs with Hybrid and multi-cloud capabilities.

I am looking forward to an exciting day tomorrow with some amazing speakers.

Thursday, 19 May 2022

Azure Synapse Influencer program

There is an exciting new program from Azure Synapse called influencers








The Azure Synapse Influencer program recognizes Azure Synapse Analytics community members who actively support one another by sharing Synapse-related content, announcements, and product news via social media.

The benefits of the new program

  • Official recognition for your support of the Azure Synapse community
  • An exclusive badge for each membership tier
  • Azure Synapse Influencer spotlights and visibility through official Synapse channels
  • Exclusive access to the Synapse engineering team, including events and Q&As
  • Content collaboration opportunities with other Azure Synapse Influencers and the Synapse team
It is great to see 3 different tiers of membership.  This is great to add to the content that Data Toboggan helps bring to the community. 

Sign up here.

Follow @Azure_Synapse on Twitter . Share Azure Synapse-related posts and content using the hashtag #SynapseInfluencers and don't forget #datatoboggan for community content

Tuesday, 10 May 2022

Tech in Ten, On the Road by Venturi!


I was excited to be a part of the Tech in Ten podcast by Venturi discussing SQLBits, Data Governance and Data Toboggan. It has been released as Episode 9.

The Tech in Ten team went down to the SQLBits data platform conference held at ExCeL London. Ben sat down with some of the speakers and data professionals at the event to talk about all things data and the ever-changing industry.


Listen to the podcast on Spotify. 

Thursday, 5 May 2022

CDO and Data Leaders Global Summit

Today was the CDO and data leaders global summit hosted by the EDM Council and CDO Magazine designed specifically for senior and C-level data and analytics executives. 
 

One of the sessions was about the EDM. 


There were discussion about the Cloud Data Management Capabilities (CDMC) 14 key controls and Automations. These should help build trust and confidence in the industry. 













The EDM Councils areas of Advocacy 

  • Best Practice - DCAM & Cloud CDMC, Data ROI, ESG Data (environmental, social & corporate governance)
  • Driving Standards - Knowledge Graph, Industry Ontologies, shared lab
  • Training and Certification - virtual & elearning and webinars and events
  • Research and Benchmarking - Global Industry Study, Life Sciences, Data Sharing
  • Regulatory Engagement - Regulators participation
  • Networking - Data Visions, CDO Summit, workgroups and forums

Learn more about the CDMC, and download with a free license for internal use, at: https://edmcouncil.org/page/CDMC

For  those who have been very DAMA Book of Knowledge focused the DCAM (Data Management Capability Assessment Model) is closely aligned. CDMC took the DCAM framework and focused on managing data in Cloud.  It contains a number of best practices specific to the challenges (and opportunities) of Cloud. The council has been in regular communication with global regulators about CDMC, on what controls may be needed in ensuring safe and effective Cloud environments.

Controls that will be used in Snowflake to satisfy the EDMC's CDMC framework can be found at the GIT link is: https://github.com/Snowflake-Labs/EDMC-CDMC-v1-14-Control_Mapping . On the Microsoft side in Microsoft Purview Compliance Manager there are templates for CDMC key controls.

The keynote was interesting talking about how the CDMC - Cloud Data Management Capabilities Framework - will have a global impact as the best practice standard for cloud Data.  The session labelled as global industry standard for accelerating trusted cloud adoption created through global industry workgroup between May 2020 to September 2021.  There were 100+ companies and 300+ participants. The panel was made up of 


 

Two reports to read
EDM Council ESG Corporate Reporting Entities report 
EDM Council ESG Rating Providers and Data Aggregators report