Microsoft Purview Ignite updates

During Microsoft Ignite there were a number of Microsoft Purview updates. 

Names changes in products were

• Microsoft Purview Data Catalog is changing its name to Microsoft Purview Unified Catalog. The goal is to provide a platform for data governance and to drive business value creation in your organization.  Microsoft mentioned data governance isn't just a gatekeeping function but when practiced well it also accelerates data value creation. 
• Microsoft Purview AI Hub was renamed to Data Security Posture Management for AI (DSPM for AI) which is now GA. 

Data Security Posture Management for AI

Data assessments (in preview) to identify and fix potential oversharing risks in your organization.

Recommendations and policies to detect risky AI usage with insider risk management policy.

Extending support for other generative AI apps, including Copilot Studio, ChatGPT - Microsoft Purview integration with ChatGPT Enterprise Compliance API

Microsoft Security Copilot in Purview is in activity explorer and it can help with data hunting and generation of filters from natural language prompts

Introducing the Model for Responsible Innovation

The Responsible Technology Adoption Unit have published a model for responsible innovation as part of DSIT It looks at the practicalities of ethical risks associated with AI or data driven projects. This model has been tested over many years originally being built in 2021 by the Centre for Data Ethics and Innovation.  Trustworthiness is at its heart with eight fundamentals areas that teams should strive for to build trust: 

• Transparency
• Accountability
• Human-centred value
• Fairness
• Privacy
• Safety
• Societal wellbeing

Read more here

Responsible Technology Adoption Unit Blog

The year ahead for SQL Server 2025

In the session SQL Server 2025 An enterprise AI-ready database platform the  core features for the next version of SQL Server were shared being based on

• AI built-in
• Best in class security and performance
• Fabric and Azure Arc connected. 

SQL Server 2025 is built from the ground to cloud with AI  ready applications. SQL Server 2025 is to have 

• Vector store - native vector data type and DiskANN index
• Model Management - model interfaces ground to cloud using REST
• Embeddings built-in - text chunking and built-in embedding generation
• Simple semantic searching - vector distance and vector search
• Framework integration - LangChain, Semantic Kernel, Entity Framework Core

All changes made for SQL will be deployed across all deployment options and all versions. 

It is integrated with Microsoft Fabric

The year ahead for SQL Server: Ground to cloud to fabric

Responsible AI is embedded and available on premises or in the cloud.

There is now SQL database in Fabric

Azure Arc extends to Azure services to SQL Server anywhere to streamline migrations to Azure. 

Watch more about the latest release from Joe Sack, Erin Stellato and Bob Ward. So many innovations in this version of SQL Serer it changes what options are available for use.

Microsoft Ignite 2024 keynote

The Microsoft Ignite keynote in Chicago November 2004 looked at adoption, extensibility and ROI. Copilot is the UI for AI. There were many announcements as expected for Microsoft Ignite

Copilot Action - regain focus time and simplify you workday Read More: aka.ms/CopilotActions

Agents in Microsoft 365 Read more:  aka.ms/AgentsinM365

Copilot studio - Autonomous agents and thirty Party Agents. Importantly Copilot Analytics including 

• Copilot Dashboard
• Business Impact Report
• Advanced Insights

to measure AI impact on your business. This impact report will help drive valuable insight to gain business value. It shows how AI is driving revolutionary productivity and changing business.

Read More: aka.ms/CopilotAnalytics

It was exciting to see the announcement of SQL Server 2025

Apply for the preview for the enterprise AI-ready database

Fabric Databases Accelerate app innovation with an AI-powered data platform with Fabric Databases .

It is a new class of cloud databases

New centralized management and governance controls and risk and safety evaluation for images with Azure AI Foundry announced

hashtag

These governance controls are in private preview

• create AI reports to govern AI apps and models
• view and manage key subscription details
• simplified setup of hubs, projects and resources

The breadth of Azure AI model was shown with their being offerings from a wide collection of frontier and open models.  Being around 1800 models currently.

Risk and safety evaluations for images to assess AI models and apps for harmful outputs. Looking to 

• Understand frequency and severity of risks
• Streamline risk assessment processes
• Measure impact of targets mitigations

Data Governance also took a central conference place because in the age of AI, Data Governance takes on an even more critical, central & important role. Watch the session 'Secure and govern your data estate with Microsoft Purview' BRK317 that covers the new security and governance capabilities. 

Successful AI transformation starts with a security-first approach. The new capabilities announced 

Prevent data oversharing and detect insider risks in Microsoft 365 Copilot

• Data Security Posture Management (DSPM) for AI, formerly known as Microsoft Purview AI Hub, is now generally available
• Microsoft Purview DLP for Microsoft 365 Copilot, now in public preview. It enables processing files based on their sensitivity label to restrict Microsoft 365
• Detect and investigate risky AI usage for Microsoft Purview Insider Risk Management and Communication Compliance in public preview

Secure and govern custom AI with new capabilities in Purview, Defender, and Azure AI

• Public preview of Microsoft Purview DSPM for AI, Insider Risk Management, and data compliance controls for agents  in Copilot Studio
• AI-Security Posture Management (AI-SPM) in Defender for Cloud is now generally available
• New centralized management and governance controls and risk and safety evaluation for Azure AI Foundry 

And last but not least  AI regulations preparation, such as the EU AI Act with new capabilities in Purview Compliance Manager and Azure AI.

• There are 4 new generally available AI compliance assessment  including the EU AI Act, NIST AI RMF, ISO 42001, and ISO 23894. 
• AI reports in Azure AI in private preview

Further Reading

Data Factory Announcements at Ignite 2024 Recap

Accelerate app innovation with an AI-powered dataplatform

What’s new with AI Azure at MS Ignite

Accelerate AI adoption with next-gen security and governance capabilities

Lots of Purview updates https://lnkd.in/eMFMx_xv at MSIgnite

Safely activate your data estate with Microsoft Purview

The next wave of Azure innovation: Azure AI Foundry, intelligent data, and more

Announcing SQL Server 2025

Delivering 3 trusted platforms for the AI age

Microsoft MVPs and Regional RDs wall 2024

I am really excited to be on the Microsoft MVPs and Regional RDs wall at Microsoft Ignite.

There are lots of amazing announcements this year for Microsoft Fabric, SQL Server, AI Copilot and Microsoft Purview to name but a few. The Microsoft Ignite 2024 Book of News is full of innovations to enable businesses to be AI Ready

Microsoft Ignite 2024 Book of News

Microsoft Ignite is the annual event that showcases technological updates and changes the way people and organizations approach work.

There were an array of  new products and key updates and the ones tha interest me the most are the  growth of Microsoft Copilot,  SQL Server 2025, Fabric databases and governance and responsible AI features.

The book of news is here.

Data Toboggan Winter Edition 2025

We are excited to announce that the winter edition, our main Data Toboggan conference,  has Call for Speakers open. This will be our 5th year so will be really exciting for us to reach that milestone.

This is our multi-track 12 hours conference to cover a many time zones.  Help us make this a really special event. 

CfS: https://bit.ly/DT2025-cfs

Event Date: Saturday 25 January 2025  

If you  want to watch recordings from our summer conference or others,  Data Toboggan Cool Runnings 2024 now has 68 videos on You Tube

Playlist: https://www.youtube.com/playlist?list=PLKp6nE2AuD_PNDt9onJGMlZxOOi8y-mPV

 

PASS 24 Data Community Summit

This week it is PASS 24 Data Community Summit

The Microsoft Keynote was entitled, Fuel AI innovation with Azure Databases.

The session abstract: Data is the fuel for innovation, driving the development of transformative AI applications. In this keynote, join Shireesh and the Microsoft engineering team as they explore harnessing the power of data across the Azure databases portfolio to drive ground breaking advancements. Take your first step on the journey with vector search, multi-agent apps and more and learn to unlock new patterns like retrieval augmented generation (RAG).  Explore the latest database innovations in SQL Server, Azure SQL, Azure Cosmos DB, Azure Database for PostgreSQL and Azure Database for MySQL that enhance operational efficiency, deliver personalized user experiences, and revolutionize the way we interact with technology. 

There were some great announcements 

Announcing

• SSMS 21 and Copilot in SSMS
• Azure SQL Database Hyperscale enhancements (Storage up to 128TB (GA) Log throughput rate up to 150 MB per second (Preview) , Continuous priming (Preview))

Preview

• Next gen general purpose on Azure SQL Managed Instance (32TB of storage, 500 DBs, Lower storage latency, improved storage importance, customisable I/O performance)
• Migration Assessment from SQL Server enabled by Azure Arc
• Mirroring in Fabric for Azure SQL Database and Azure Cosmos DB
• In-database embedding generation
• Vector Indexing and search in Azure Cosmos DB for NoSQL
• Native vector type and functions in Azure SQL Database (Available now)
• DiskANN indexing in Azure Database for PostgreSQL

Generally available

• Bi-directional disaster recovery with link feature in Azure SQL Managed Instance
• Flat and quantized flat vector indexes in Azure Cosmos DB for NoSQL
• Serverless auto-pause delay to 15 min
• Azure AI extension
• Hyperscale elastic pools
• Azure Cosmos DB for MongoDB enhancements

Data Toboggan Alpine Coaster 2024

Alpine Coaster is our shorter 3 time zone end of year unconference event. You know we like to mix things up with this version of Data Toboggan, so this time:

Sessions are around 20 minutes long.

There are three session editions to join. APAC 7:30 8:40 | EMEA 12:30 - 1:40 | AMER 5:30 - 6:10 GMT, join one or all

Please note, there will be no recordings taken on the day. We feel that kinda takes the pressure off a little for this short session more interactive type of event.

Register now for the Alpine Coaster event: Friday 8 November 2024

APAC Edition - https://www.meetup.com/data-toboggan/events/304285946/

EMEA Edition - https://www.meetup.com/data-toboggan/events/304285953/

AMER Edition - https://www.meetup.com/data-toboggan/events/304285956/

Microsoft Purview Data Catalog billing needs consent

There are changes to Microsoft Purview Data Catalog billing beyond 1 November 2024. 

NOTE: The new pricing model for Microsoft Purview Data Governance now goes into effect on January 6, 2025.

The article on Microsoft Purview Data Catalog billing consent explains the new pricing model for Microsoft Purview Data Governance. Users who have been using the data catalog for free will need to consent to the new pay-as-you-go billing model to continue using the service beyond this date. The consent process involves accepting the terms and conditions through a pop-up in the Microsoft Purview portal.

If the necessary consent is not provided by the Entra Global Admin, data governance concepts created in the unified portal will be retained for 120 days. The roles required to provide consent include the Microsoft Entra Global administrator or the Microsoft Entra Compliance administrator role, along with the DataMap curator role.

Read more about it here

https://learn.microsoft.com/en-us/purview/purview-portal-billing-consent

Microsoft Purview Data Loss Prevention policies have been extended to Fabric lakehouses

Microsoft Purview's Data Loss Prevention (DLP) policies have been extended to Fabric lakehouses and are now in public preview. This extension follows the success of DLP for Power BI and the general availability of Microsoft Fabric last year1. DLP policies help automatically detect sensitive information as it is uploaded into lakehouses and take risk remediation actions to comply with regulations like GDPR and HIPAA. Compliance and Security administrators will receive audit logs for every detection and can set up alerts for sensitive information found in their Fabric lakehouses. There will be no charge for lakehouses scanned by DLP policies during the preview period.

You can read more here

Microsoft AI Tour London 2024

The Microsoft AI Tour in London on 21 October 2024 talked about solving the biggest challenges and uncovering new opportunities to empower your organization. The Opening Keynote was with Clare Barclay, Satya Nadella and Jared Spataro.  There were discussions on building an agentic world where  personal agents, organisational agents, business process agents and cross organisation agents enable the use of AI in business to assist you with work. An example of this is Pages for AI artifacts that enables you to work with the organisation and AI . The 3 areas of innovations

• Copilot
• Copilot & AI Stack
• Copilot devices

These innovative areas will increase value and reduce waste and Copilot Studio for autonomous agents was announced. 

A very important announcement, for us in the UK, is that Microsoft and UK Government have signed a five-year agreement to support the new era of digital transformation. So a very exciting time for the UK to grow.

There is gravity with data. AI needs to bring all the data together in one platform like Microsoft Fabric, to simplify data and have App servers, AI apps and tools. Microsoft showed its unified data offering in IaaS, PaaS and SaaS to unify the data estate, at what ever stage a business is at, for databases and analytics. 

Trust is the most important thing in #AI. You must have a set of principles considering Security, Privacy, Safety at the core

AI is redefining Britain. It can take longer to realise value as change and adoption management takes time. It is cheaper to innovate than before so we need to think big, think bold and be ambitious 

Satya finished with 4 important points to note: expand opportunity, earn trust, protect fundamental rights and advance sustainability.

The announcement today can be read here

Autonomous agent capabilities across Copilot Studio and Dynamics 365 to help scale the impact of every individual, team, and business function. 

What's new for Data Toboggan

There are plenty of things coming from Data Toboggan.

Videos from the last conference are being published on YouTube 

Alpine Coaster call for speakers is open https://sessionize.com/data-toboggan-alpine-coaster-2024/

The Winter conference date to add to your diaries: 25 January 2025.

European Microsoft Fabric Community Conference Keynote

The Microsoft Fabric Community Conference is taking place this week in Stockholm, Sweden. There are 3300 attendees which is amazing.  The Keynote was entitled Microsoft Fabric Vision & Roadmap – Analytics in the Era of AI.  The keynote speakers; Arun Ulag, Kim Manis, Amir Netz, Marco Casalaina, Wangui McKelvey and Rudra Mitra from Microsoft.

The announcements blog contains a vast number of innovative enhancements and capability updates. The blog is: European Fabric Community Conference 2024: Building an AI-powered data platform

Fabric is about providing a unified data platform for AI transformation with these 3 areas

• AI-powered development: Fabric can give teams the AI-powered tools needed for any data project in a pre-integrated and optimized SaaS environment.
• An AI-powered data estate: Fabric can help you access your entire multi-cloud data estate from a single, open data lake, work from the same copy of data across analytics engines, and use that data to power AI innovation 
• AI-powered insights: Fabric can empower everyone to better understand their data with AI-powered visuals and Q&A experiences embedded in the Microsoft 365 apps they use every day. 

Microsoft Fabric is shown in the AI Copilot stack.

There is a new exam available to enhance skilling in data engineering DP-700 This Certification measures Fabric data engineer skills:

• Implementing and managing an analytics solution
• Ingesting and transforming data
• Monitoring and optimizing an analytics solution

Bringing all the skills together for data engineering, data science, and data analytics. You can read more here

 There  are numerous technical announcements

You can see the list of GA features shipped for network security, data security and governance. There is an End-to-End Security Whitepaper to the read aka.ms/FabricSecurityWhitepaper 

The public preview of the new copy job item provides

• Built-in support for copying data between workspaces
• Streamline copying data into and across OneLake
• Setup automatic incremental copy

To help with deployment, the public preview was announced of the Terraform Provider for Microsoft Fabric. This will help empower deployment, management, and governance. This Microsoft Fabric Infrastructure as Code (IaC) new Terraform Provider lets you use the same tools and processes already employed for Microsoft Azure and other cloud resources.

• Check out the provider on the Terraform Registry: https://aka.ms/FabricTF
• To help you get started, there is a handy Quick Starts repository with examples: GitHub Examples for using the Terraform Provider for Microsoft Fabric

Access Azure Databricks Unity Catalog tables directly from Fabric via the new Mirrored Azure Databricks Catalog feature, now in Public Preview https://blog.fabric.microsoft.com/en-US/blog/databricks-unity-catalog-tables-available-in-microsoft-fabric/

A new feature in Fabric: Lakehouse schemas https://blog.fabric.microsoft.com/en-us/blog/organizing-your-tables-with-lakehouse-schemas-and-more-public-preview/

Public Preview of T-SQL Notebook in Fabric  https://blog.fabric.microsoft.com/en-us/blog/announcing-public-preview-of-t-sql-notebook-in-fabric

The General Availability of Copilot for Data Factory in Microsoft Fabric  https://blog.fabric.microsoft.com/en-us/blog/announcing-the-general-availability-of-copilot-for-data-factory-in-microsoft-fabric

All the in dept announcements are on the Fabric Blog. 

The unified data culture was mentioned between Fabric and Office connected through Power BI. 

There are lots more releases out there. This is such an exciting time in the Data and AI space. The exciting announcement is that the conference will be back next year. 

Microsoft Purview at the Fabric Conference

There are a  number of  Microsoft Purview sessions at the European Microsoft Fabric Conference this week in Sweden. 

Check out the schedule 

Good Morning European Microsoft Fabric Community Conference

 Good Morning for day 2 of the European Microsoft Fabric Community Conference

Microsoft Security Community

Microsoft Purview helps in the world against cyber attacks. There is a security community that you can join for updates.  This blog lists some of the useful upcoming webinars in the Microsoft Purview space. 

SEP 18 2024 Microsoft Purview | Microsoft Purview eDiscovery Modern UX

Discussing the new Purview eDiscovery UI, walk through the new features and discuss how the new UI has been optimized and how the new enhanced features that have been added to help simplify and boost user experience.

OCT 15 2024 Microsoft Purview | Navigating the Future of AI: Understanding the EU AI Act and Its Implications

The European Union Artificial Intelligence Act (EU AI Act) is a pioneering regulatory framework designed to oversee the development, deployment, and use of artificial intelligence (AI) technologies across the EU. This landmark legislation aims to ensure AI systems are safe, ethical, and transparent, prioritizing the protection of fundamental rights and user safety.

OCT 16 2024 Microsoft Purview | Introduction to Data Lifecycle Management (DLM) | Compliance Manager | Part 1

Data Lifecycle Management (DLM) is a comprehensive approach to managing data throughout its lifecycle, from creation and storage to archiving and deletion. DLM in Compliance Manager involves a structured approach to managing data through its entire lifecycle within the framework of regulatory and organizational compliance requirements.

Microsoft Purview Data Governance GA Rollout

The new Microsoft  Purview solution is currently being rolled out.  The details on when the region will be available is here: New Microsoft Purview Data Catalog deployment regions (Preview) | Microsoft Learn. 

This new updated tool is critical to make sure data is AI ready across the business. There is a good immersive tour of the Purview Data Governance solution: https://purviewdatagovernance.storylane.io/share/0fu0nmhvphk4

Microsoft Purview: Data Governance Solution is General Availability

The new Microsoft Purview: Data Governance Solution became General Availability 1 September.

This release brings with it a several new capabilities that have added since the public preview launched in April 2024.  This new release focuses on the practice of responsibly federated data governance in built in the application.  The release enables

• modelling and use of durable business concepts to govern evolving and multi-cloud physical data and AI estate
• scale the management of data quality and health in the data estate with the practice of federated data governance, with intelligent automation, and without displacing the people interaction 
• confidently activate responsible discover, understand, and access data in the organization-wide teams and enable users help innovate.

Microsoft Fabric Known Issues

There is a Power BI report that contains the list of known feature issues for the various areas of Microsoft Fabric. The report can be found here.

The AI Hub in Microsoft Purview

The Microsoft Purview AI hub is a preview feature that helps you govern and gain insight to generative AI apps and helps protect through mitigating and managing risks. With the explosion of generative AI this has become a crucial addition in the governance world.

The AI Hub helps with

• strengthening information protection for Copilot
• supporting compliance management for Copilot
• fortifying data security for AI
• insights and analytics that enable that view on AI activity
• policies to help protect and prevent data loss
• compliance for optimal data handling

The use of Purview AI hub works well with other capabilities to improve data security.

• Sensitivity labels and content encrypted by Microsoft Purview Information Protection
• Data classification
• Customer Key
• Communication compliance
• Auditing
• Content search
• eDiscovery
• Retention and deletion
• Customer Lockbox

The insights, policies and control secures data from Microsoft Copilot for Microsoft 365 and AI apps from third-party large language modules (LLMs).

The changing new Microsoft Purview Portal

Microsoft unveiled the new Purview portal, a game-changer for organizations navigating the complex landscape of data security and risk management. This change is ongoing within Microsoft Purview with the relocation of portal features and retirement of some features. The new Microsoft Portal features changes

The portal's design facilitates easy navigation with the solution cards for quick access to the solutions within  Risk & Compliance, Data Governance, and Data Security sections.

The key features areas are now

Data Security: covers discovery and protecting sensitive information across your organization with a robust set of data security solutions, including Data Loss Prevention and Information Protection and adaptive protection .

Data Governance: It offers unified data governance solutions to manage services across your on-premises, multi cloud, and SaaS environments. This includes creating an up-to-date map of your data estate with classification, data catalog, data quality (preview), lineage, data management, and data estate insights.

Risk and Compliance: The platform includes solutions to help minimize compliance risks and meet regulatory requirements, ensuring your organization stays ahead of the curve in data protection standards with eDiscovery and audit, communication compliance, data lifecycle management, and records management.

The Growth of Microsoft Purview

It has been an interesting few years watching the growth of a data governance solution into a product that truly helps govern business data. The growth of  Purview has seen many changes over the last few years, to its current incarnation that becomes GA in September. The addition of in-product AI, the Responsible AI tooling, coming in the AI Hub, adds a new dimension, which will enable the next evolution for data governance.  The integration of Microsoft Fabric and Microsoft Purview will help analytics solutions have better quality data out of the box.

hashtag#MicrosoftPurview