Microsoft Purview Fabric announcements

There were a number of announcements at the Microsoft Fabric Community Conference including the new Microsoft Purview for modern data governance was shared.  With business moving towards federated governance models, managed by line of business to help with more local understanding and increasing volumes of data, Microsoft have launched in Purview the capability for organizations to create subdomains to refine the way the data estate is structured in Fabric. Security has also become easier with the ability to set security groups for default domains. 

Microsoft Fabric is now natively integrated with Microsoft Purview Data Governance solution. There is a reimagined data governance experience for the data estate governance practice. The new experience includes data curation, an important new feature including data quality with insights. The new experience is available in preview 8 April 2024. This new experience is aiming to help accelerate measurable business value with key results, simplification and to help with implementing efficiency with natural language recommendations. 

Purview enables business terminology linkage to 

• Data Products (a collection of data assets used for a business function) 
• Business Domains (ownership of Data Products) 
• Data Quality (assessment of quality) 
• Data Access, Actions 
• Data Estate Health (reports and insights)

A really exciting new feature we have all been waiting for is the data quality capabilities.  The is now the Data Quality model to set rules top down with business domains, data products, and the data assets. The model generates data quality scores at the asset, data product, or business domain level from the policies on terms or rules.  The score rules show on the dashboard as red/yellow/green indicator scores. The 2 capabilities in this data quality model are:

• Profiling—quick sample set insights 
• Data quality scans—in-depth scans of full data sets

It is great to see the Microsoft Purview continues to align to the EDM Council set of 14 rules. 

There is now an actions centre showing the current health summarising actions by role, data product or business domain for governance. This actions centra aims to help improve governance posture for the business. 

There is partnership with Ernst & Young LLP who will share playbooks and reports for US financial services customers on Azure Marketplace, throughout the preview. 

In summary there is a shift away from traditional IT-centric data architecture to federated architectures such as data mesh. The automated way to deal with Data Quality is a game changer for business. 

References

Announcements from the Microsoft Fabric Community Conference

Easily implement data mesh architecture with domains in Fabric

Introducing modern data governance for the era of AI 

The foundation for responsible analytics with Microsoft Purview

Watch: The Unified Data Platform for the Era Of AI | Microsoft Fabric Community Conference Day 1 Keynote

Crash Course in Microsoft Purview (azureedge.net)

Learning

• Sign up for the Microsoft Fabric free trial.
• Visit the Microsoft Fabric website. 
• Join the Fabric community. 
• technical blogs on the Microsoft Fabric Updates Blog. 
• Explore the Fabric technical documentation. 
• Join the Fabric Partner Community on Teams to engage with the Fabric partner and product teams and attend weekly Fabric Engineering Connection calls every Wednesday at 8 am PDT.  

Responsible AI dashboard training

There is a new MSLearn course to Learn how to debug an AI model using the Responsible AI dashboard in Azure Machine Learning studio to ensure it performs responsibly and is less harmful. It is important to understand and learn how to use the dashboard to set any projects up for success.

Train a model and debug it with Responsible AI dashboard

The objectives are 

• Create a responsible AI dashboard.
• Identify where the model has errors.
• Discover data over or under representation to mitigate biases.
• Understand what drives a model outcome with explainable and interpretability.
• Mitigate issues to meet compliance regulation requirements.

You do need the ability to understand beginner level Python.

The Fabric Conference 2024

The first Microsoft Fabric Community Conference, took place from 26 to 28 March 2024, at the MGM Grand in Las Vegas, Nevada.  It was an in person only conference and no sessions were recorded or streamed.  Great to see so many back to in person conferences, although for those not able to attend it means limited learning. 

The conference had more than 130 sessions covering various aspects of Microsoft Fabric from data warehousing to data movement, AI, real-time analytics, and business intelligence.

The Microsoft Intelligent Data Platform incorporates Microsoft Fabric, a suite of technologies that empowers organizations to harness the full power of their data. By natively integrating products across four critical workloads AI, analytics, database, and security, organizations can innovate without limits. The great advantage of fabric is that it brings together disconnected services from multiple vendors to  focus on accelerating transformation.

The four core promises of Fabric:

• Fabric is a complete platform
• Fabric is lake-centric and open
• Fabric can empower every business user
• Fabric is AI powered

There was a huge number of announcements that represent just the start of the innovation to Microsoft Fabric platform.  The full set of announcements are here. I will share separately about all the Purview announcements as these will add the depth we need to drive forward with AI. A number of other features I will blog about separately as they change how Fabric is growing. 

Announcements from the Microsoft Fabric Community Conference

Mirroring in Fabric is a great addition to help with the data warehouse journey to Fabric.  
Announcing the Public Preview of Mirroring in Microsoft Fabric

Azure Private Link support for Microsoft Fabric 

Create folders and sub folders in workspaces and being able to tag Fabric items in futures will be a huge plus for compliance.
Announcing Folder in Workspace in Public Preview

Microsoft Fabric has a release plan that is documented.

Next events to look out for

Microsoft Build from 21-23 May 2024 is either in person in Seattle, Washington, or online. 

PASS Summit Community Conference  4-8 November 2024

Excited that there will be a second Fabric Conference next year 1-3 April 2025 at MGM grand, Las Vegas https://aka.ms/FabCon25

Responsible AI Day at Microsoft

Today I took the opportunity to attend a Microsoft UK  Partner Responsible AI day at TVP in Reading. Thank you to Robin Lester and the RAI team for putting on an informative day of sessions.  I also got the opportunity to speak to Claire Dugan, a Responsible AI Advocate at Microsoft UK to discuss Governance and AI.

There are several places to get started with learning about the tools that are available to create impact assessments for projects.

FOUNDATION GUIDES 

• The Responsible AI Maturity Assessment - (RAI MM) is a framework to help organizations identify their current and desired levels of RAI maturity
• Responsible AI Standard v2 -  Microsoft’s six AI principles operationalized
• Responsible AI Impact Assessment Template - To define a process for assessing the impact an AI system may have on people, organizations, and society.
• HAX Playbook  -  The HAX Playbook is a tool for proactively and systematically exploring common human-AI interaction failures 
• Overview of Responsible AI practices for Azure OpenAI models
• Azure AI Content Safety

It is necessary to adopt principles to create safe and explainable systems to ensure fair, transparent and safe systems are designed and deployed.  More details can be found

Empowering responsible AI practices 

The SQLBits 2024 Event

SQLBits 2024 was amazing as ever. The organisers creating another well choreographed event. It was held in Farnborough, the birthplace of British aviation. 

There was a huge number of tracks including all types of sessions.  

The Sessions 

The agenda covered various types of sessions

• Tuesday Training Day
• Wednesday 100 minute sessions to gain more depth into a variety of areas
• Thursday General Sessions Day One
• Friday General Sessions Day Two
• The Free Saturday

SQLBits Extra Events

There was a wide range of extracurricular things to get involved with to enable a different slant on networking:

• Meet the Trainer Monday 18th March, 6.30pm, The Aviator Hotel
• Welcome Drinks and Burgers & Board Games Night Wednesday 20th March, 6pm
• Ask the Experts Wednesday 20th March - Saturday 23rd March
• The SQLBits Run Wednesday 20th March, 6pm & Friday 22nd March, 6am
• User Group Bonus Sessions Thursday 21st March, 6pm
• The Pub Quiz Thursday 21st March, 7.30pm
• The Friday Night Party Friday 22nd March, 7.30pm

The Keynote

This was delivered by a number of speakers.

SQLBits announcements

Public Preview:  Managed Instance General Purpose Next-Gen; Migration Assessment in Azure Arc; Database Watcher.

Private Preview : T-SQL Regex; Copilot in Azure SQL Database

Learn more: 

Introducing Azure SQL Managed Instance Next-gen GP

Introducing database watcher for Azure SQL

Azure SQL migration assessment enabled by Azure Arc

Introducing Copilot in Azure SQL Database

SQLBits Buddies 2024

I was part of a team of helpers at SQLBits who are Bits Buddies. We are all experienced helpers and have attended lots of SQLBits. We are dedicated to help attendees who might want a bit of extra company and support, whether it’s a persons first time at the event or a regular attendee at the event.

We ran pre event meet up opportunities in the run up to SQLBits. The weekly drop ins for delegates and those interested in attending for an informal chat about the experience of attending SQLBits and to make connections before the event. It was nice to meet a few people before the event. This year the bits buddies wore orange hats so we could be seen easily around the venue.  It was really nice to speak to people at the event and help with questions. Till next year.

Data Toboggan Slide Preparation at SQLBits

This year SQLBits Thursday 21 March 2024 , added User Group Bonus Sessions.  It was announced as

After the main sessions, two UK user groups are running sessions that you’re welcome to join:

-  London Fabric User Group - SQL Bits Special - Ask Me Anything Panel (Gate 4)

-  Data Toboggan - Ask the Fabric Experts (Gate 1)

Running from 18:00 - 19:00, each with a panel of experts ready to answer questions or discuss hot topics.

Data Toboggan Slide Preparation ran its FIRST in person User Group hosted by Richard Munn at SQLBits.

The panelist were  Richard Munn, Dr Victoria Holt,  Cathrine Wilhelmsen, Mark Pryce-Maher, Emilie Rønning and Andy Cutler taking questions from the audience.

James Reeves reported 

'Data Toboggan User Group Celebrates First In-Person Meetup

🎉 The Data Toboggan user community recently celebrated an exciting milestone: their first-ever in-person meetup. After connecting and collaborating online, members finally had the chance to gather face-to-face and connect with fellow data enthusiasts who share a passion for uncovering insights through data.

💡 A highlight of the gathering was a session focused on Microsoft Fabric, a comprehensive analytics and data platform. Attendees engaged in a lively discussion about how tools like Microsoft Fabric are revolutionizing the field of data analytics and shaping the future of the industry.

🙌 The organizers expressed their gratitude to everyone who made the meetup possible and to all who participated. The energy, enthusiasm, and sense of community at the event were truly remarkable. They look forward to more opportunities for the Data Toboggan user group to connect, both virtually and in person.'