The second part of the governance story.
"The important thing is not to stop questioning. Curiosity has its own reason for existing" Einstein
Wednesday 28 December 2022
Wednesday 14 December 2022
In the Azure Synapse Analytics month's MVP series video, Ryan Majidimehr was joined by two MVPs, myself and Andy Cutler, to talk about the upcoming Azure Synapse conference called Data Toboggan. https://msft.it/6019eRKad
Friday 9 December 2022
A tale for December. The business and the frameworks.
Monday 21 November 2022
Thursday 17 November 2022
The keynote is about Doing More with Less: The Challenges Ahead for Every Data Professional. These are challenging times for every business everywhere. So how can you ensure you get the most from your investment in IT, to meet your needs now, while also preparing you for an uncertain future? Join Jakub Lamik, Steve Jones and Kathi Kellenberger in this keynote which looks at how the world is changing for data professionals, and the areas to focus on which will bring the best return on your investment in the long term.
Steve Jones | Advocate
Jakub Lamik | CEO
Kathi Kellenberger | Customer Success Engineer
David Bick | Head of Product Marketing
Arneh Eskandari | Solutions Engineering Manager
There were123 sessions yesterday. Make sure you take advantage of the networking opportunities.
16.10 Steve Jones opens the keynote talking about Kalen the technical people he had met. Opportunity, to learn and be inspired.
Redgate are expanding across multi databases with there tools. Redgate live the Devops way, build a great culture internally and community centred. SQL Saturday started to bring it to the community. Redgate gave it to PASS. Then donated PASS away after buying it last year.
The CEO of Redgate is on the stage. More than a conference: a homecoming. Why important to invest in the community? Database professionals answer important questions about our world.
Every business is a software business - Satya Nadella
Software means data really, how data is ubiquitous and how it enables transformation. Change is constant.
How has the database landscape changed.
Open source is increasingly important, all sectors and all segments. A powerful tool for community.
Cross database estate is on the rise. - From the state of the data estate report.
Fragmentation is accelerating. Developing are influencing the databases they use.
PostgreSQL and MySQL are developing the most. SQL Server in the top 6.
Multi database environments are here to stay and complexity is growing - Steve Jones
Micro Services enable agility between software and dev teams - further drives fragmentation
Complexity is mirrored in the environment and automation required with a standard set of tool and approach. This helps with the Devops environment.
David Bick is on the Stage
Just solving SQL Server problems is just not enough for our customers. A lot of postgreSQL growing.
Flyway is a database schema for a standard approach using more than 25 databases. From on prem to cloud. Redgate bought and support the open support project.
Gives richer automation. Design principles - should be able to work the way you want to, flexibility, Tools and approach need to be the same but the changes need to be database specific.
Flyway Enterprise is released for SQL Server and Oracle. This will unlock devops for developers.
Migration to cloud is expanding and Redgate can deploy in hours. Cloud required collaboration. No more silos. DBAs must be database agnostic and support hybrid database estates.
Arneh is on the Stage. Security and monitoring is centralised, but individual business units chose different platforms.
Scalability and being held back . Domain expertise is needed from various cloud technologies. Not enough dashboards across all of these. Cost control is required to be kept as low as possible.
A typical financial set up 1000+ database 50/50 split on prem and in the cloud in the UK and US. They need granular monitoring as there is no holistic monitor.
About solutions, community and support.
How to help you grow. There are lots of new speakers at Summit this year. How many children want to be a DBA when they grow up. Simple talk is a great launching point for your career. Louis Davidson is the new editor of simple talk.
Adoption, results and advocacy in customer success. They are the face of the customer internally and getting value out of there products. Customer success is who Redgate are. Also further ongoing professional development. There are free industry learning classes which were created in the pandemic.
The debut Redgate 100 most influential in the database community 2022 was shared.
The day 1 keynote of the first in person PASS Data Community Summit was packed with exciting data platform innovations across SQL Server and the Microsoft Intelligent Data Platform.
My visit to PASS Data Community summit in Seattle involved my first flight out of the UK since Covid, 3 speaking opportunities and a visit to the Microsoft Reactor in Redmond to speak with the product teams. It is nice to be out travelling to conferences again and networking with the data community. There is so much added value to be had.
Monday 14 November 2022
I have 2 speaking sessions at PASS Data Community Summit on Wednesday 16 November. The live Q&A from my on demand session on:
The adoption of Data Governance is often marred with thoughts of control and security. Data Governance is about much more in the new world. It is about driving business transformation, knowing what value your data has, and how to remove those data silos, bringing with it a new data culture. In this session, we will start to explore what the new world of Data Governance is and how we help businesses understand the benefits, why it is important, and how to get started in simple agile steps. This session will help you understand why Azure Purview brings exciting features that slot into the world enabling data governance with ease. I share more about this in a blog post building data governance into everyday processes
A panel session exploring the role of the database in today’s digital transformation and modernization initiatives. What are the experiences of transformation initiatives and why the database has to be included in these innovation drives for the benefit of increased speed, efficiency and risk management. I share more about this in a blog post The two sides of innovation: Data and DevOps
Thursday 10 November 2022
We have a mini conference to close the year out called Alpine Coaster on 25 November 2022.
How did Alpine Coaster get its Name
The longest toboggan run in Switzerland is the Pradaschier Rodelbahn. It has 31 curves and a difference in elevation of 480 m. The toboggan run coasts its way down to the valley with twists and turns. It is 3,060 m long. It is the most extensive alpine coaster run in Switzerland. It can move up to 40 kilometers/hour.
It takes between 7-10 minutes to ride Switzerland's longest toboggan run, the Pradaschier Rodelbahn.
Alpine Coaster Challenge
Create a Session between 7 – 10 minutes long, so bite-size sessions for hungry data professionals. This hopefully will leave everyone all excited for our full conference 28 January 2023.
Tuesday 1 November 2022
There have been a hole raft of changes to Microsoft Purview of late which can be found on the Microsoft Purview blog announcements can be seen on the Security, Compliance, and Identity Blog
Report Manual Data lineage with few clicks in Microsoft Purview https://techcommunity.microsoft.com/t5/security-compliance-and-identity/report-manual-data-lineage-with-few-clicks-in-microsoft-purview/ba-p/3655228
Now in Public Preview: Microsoft Purview workflows HTTP connector
Best practices for Purview and a federated way of working
Catalog Adoption: Discover more with Data estate insights in Microsoft Purview
New machine learning classifiers in Microsoft Purview Governance
Data curation: Discover more with data estate insights in Microsoft Purview
Monday 17 October 2022
There were several announcements at Microsoft Ignite about Microsoft Purview.
- ML Based Classification
- Manual Lineage
- Dynamic Lineage for Azure SQL Database
- Self-service access for Azure SQL Database
The details are
Saturday 15 October 2022
To enable data to be managed well it is important to have a business lens applied to the data. To that end Microsoft have added a new capability to enrich the Data Map with business and governance context to make it relevant for the Data Stewards, Chief Data Officers, Data Analysts, and Data Scientists. This new feature is called Purview Metamodel. Three elements are organisation, business processes and data products. This shows how the data is used in business activities, enables the organisational hierarchy to be defined and , defines data used across the business departments and business processes. This is all placed this in the hands of the Data Steward.
To add further enrichment and augmentation to Purview Data Map it provides end-to-end data lineage which the Data Stewards can now annotate so their data in manual data lineage data movement is shown.
Thursday 13 October 2022
The Microsoft Ignite 2022 keynote on 12 October shared many innovations and there were lots of announcements. The Microsoft Book of News October 12 - 14, 2022 details these https://news.microsoft.com/ignite-2022-book-of-news/
Satya talked about the digital imperative and the world's computer - Azure. The main 5 themes are to do more with less.
The technology world is changing rapidly and Satya mentioned that Gartner predicts that by 2025, 70 % of applications will be made by no code/low code tools, up from 25% in 2020. enabling hand drawn forms for example to be converted into apps with AI
Microsoft and Databricks deepen partnership for modern, cloud-native analytics
Microsoft and Databricks have partnered to build a foundation in the Microsoft Intelligent Data Platform by integrating their hallmark capabilities to create an integrated solution for our customers.
Distributed PostgreSQL comes to Azure Cosmos DB
Azure Cosmos DB for PostgreSQL, a new Generally Available service to build cloud-native relational applications. Azure now offers its own single database service that supports both relational and NoSQL workloads. You can build cloud-native applications for relational and non-relational data using Cosmos DB.
Introducing the Microsoft Intelligent Data Platform Partner Ecosystem
There was a launch of a powerful new Partner Ecosystem for the Microsoft Intelligent Data Platform deliver category-leading and cloud-native data and AI solutions integrated with the Microsoft Intelligent Data Platform to complement capabilities to address diverse scenarios.
Data Governance was mentioned at every single opportunity in many sessions. Data Governance looking at security and compliance, data management and responsible democratisation. Announcements were Microsoft Purview Business workflows (GA), Business metamodel (Preview), Improved root cause analysis and traceability with SQL Dynamic lineage
More details in the book of news shares:
- Improved root cause analysis and traceability with SQL Dynamic lineage (now generally available) and fine-grained lineage (in preview) on Power BI datasets. You can do thorough root cause analysis from a single location in Microsoft Purview.
- Metamodels that will enable customers to define organization, departments, data domains and business processes on their technical data. This feature is in preview.
- then the Machine learning-based classifications will make detection of human names and addresses simple and scalable in user data. This feature is in preview.
These features with help with big data management, adding context on top of data, improved classification with AI, manual process lineage and scorecard insights. You can read about it below
Add business context to your hybrid data estate with Microsoft Purview
Customize retention and deletion to help meet your specific business requirements
Read the article to learn more about the announcements in the Data Lifecycle and Records Management space that help organizations manage the lifecycle of data.
What's New in Microsoft Purview Compliance Manager
Compliance manager helps
- Eliminating blind spots with the right set of security, compliance, and privacy controls
- Safeguarding critical data from external and internal threats
- Identifying risks and addressing regulatory compliance requirements
There has been additional automated controls for Microsoft Priva and App Governance announced.
Power BI updates Do more with enterprise self-service business intelligence https://powerbi.microsoft.com/en-us/blog/microsoft-ignite-2022-do-more-with-enterprise-self-service-business-intelligence/
with video summary https://www.youtube.com/watch?v=PxGVcFb-zv0&t=28s
Tuesday 13 September 2022
I am pleased to share my session schedule
Business Benefits of Good Governance
Summit - Day 1 09:30 AM - 10:00 AM PST
Category: Live Q&A
Theme: Ahead of the curve
Transformation and Innovation: Why the Database Must be Included
with Steve Jones and Joshua Higginbotham
Summit - Day 1 02:30 PM-03:45 PM PST
Theme: Ahead of the curve
Data Governance with Azure Purview - Ask the Experts ( Now Microsoft Purview)
With Erwin de Kreuk and Wolfgang Strasser
Summit - Day 3 09:30 AM - 10:45 AM PST
Monday 12 September 2022
Microsoft Purview has a set of predefined roles with differing levels of access defined here. There is always a lot of reading on these pages but further down I spotted a useful chart. Diagrams help show the access clearly.
Thursday 8 September 2022
Metadata is an important facet of data entities and sits within the data governance space. Meta data is data about data. There are various categories of metadata.
- Business metadata describes all aspects used for governance, finding & understanding data.
- Technical metadata describes the structural aspects of data at design time.
- Operational metadata describes processing aspects of data at run time.
- Social metadata describes the user perspective of the data by its consumers.
Often metadata connects business domains, processes, technology and data. A good place to start is at the technical layer and a tool such as Microsoft Purview can be useful to make this scalable. The end goal is understandability of the data leading to good data quality.
Saturday 3 September 2022
There are a set of solution accelerator on GitHub for Microsoft Purview. These are:
A connector to ingest Azure Databricks lineage into Microsoft Purview
Solution accelerator to help build Machine Learning Lineage
Solution Accelerator to help build Purview custom connectors
Solution accelerator for creating custom type definitions in Microsoft Purview.
Wednesday 31 August 2022
With Microsoft Purview gaining ground it is useful to find learning resources. One such resource I found is this Purview lab.
The Workshop URL https://aka.ms/purviewlab
The learning models are
Monday 15 August 2022
I came across this article giving a comparison between what is included in DAMA-DMBOK2, DCAM and TOGAF methodologies. I mentioned the core framework elements here. The most used data models by the industry are DAMA-DMBOK2 by the DAMA International and DCAM® 2.2 by the EDM Council.
No one model covers all areas and no one company is the same and it is very common that different bits are used as and when required. It is worth reading the discussion in the blog.
Wednesday 10 August 2022
Excited to receive my 5th MVP award kit containing my 5th disk to go on my MVP Crystal Award.
I am also looking forward to receiving my 5 years milestone disk as an MVP in a few weeks as well. Working with community events enabling others to get access to free training and helping others learn is so important. Here is to the next year and seeing what exciting things we can make happen.
Friday 29 July 2022
Microsoft Purview Data Sharing allows in-place data sharing for Azure Data Lake Storage (ADLS Gen2 ) and Blob Storage. This solves the problem of data proliferation where you can often end up with multiple copies of the same data resulting in higher storage costs and tracking issues from data movement. This also reduces the need for more ETL packages to manage and reduces the time it takes to get access to the data. Managing the data movement can be time consuming so this data sharing feature will help reduce this.
This new feature will enable better data management, controlled access to the data with automated data sharing. The feature works as as below:
A few things to note
- the data provider pays for data storage and their own data access
- the data consumer pays for their own data access transactions.
Access can be changed
- the data provider can revoke access to the share or set a share expiration time for time-bound access to data.
- the data consumer can also terminate access to the share at any time.
Thursday 28 July 2022
Microsoft have released a collection of community resources to help accelerate learning and enable interaction with other members of the community. Resource guide here
In addition to the Microsoft resources are those created by presenters at the Data Toboggan set of Azure Synapse conferences. Follow, share and like us.
Twitter: https://twitter.com/datatoboggan @datatoboggan
Conference Session recordings
Community Call to Action
Follow us on Twitter: @Azure_Synapse
Read and comment on our blog: https://aka.ms/SynapseBlog
Check out our monthly updates blog: https://aka.ms/SynapseMonthlyUpdate
Subscribe to our YouTube channel: https://aka.ms/SynapseYouTube
Share and vote for ideas to improve Azure Synapse: https://aka.ms/SynapseIdeas
Join the Azure Synapse Influencers program: https://aka.ms/SynapseInfluencers
Product page: https://aka.ms/Synapse
Reference architectures: https://aka.ms/SynapseArchitectures
Security whitepaper: http://aka.ms/SynapseSecurity
Technical Discussions and Q&A
Microsoft Q&A: https://aka.ms/SynapseQuestions
Get started in 60 minutes: https://aka.ms/SynapseGetStarted
Azure Synapse Analytics Toolkit: https://aka.ms/SynapseToolkit
MS Learn – Learning paths: https://aka.ms/SynapseLearningPaths
Synapse Practitioner : https://aka.ms/SynapsePractitioner
Microsoft Virtual Training Days: https://aka.ms/SynapseMVTD
30-Day Cloud Skills Challenge: https://aka.ms/SynapseSkillsChallenge
Samples & Accelerators
Samples on GitHub: https://aka.ms/SynapseSamples
Accelerator – End-to-End Analytics: https://aka.ms/azsynapsee2e-git