Welcome

Passionately curious about Data, Databases and Systems Complexity. Data is ubiquitous, the database universe is dichotomous (structured and unstructured), expanding and complex. Find my Database Research at SQLToolkit.co.uk . Microsoft Data Platform MVP

"The important thing is not to stop questioning. Curiosity has its own reason for existing" Einstein



Showing posts with label FabConEurope. Show all posts
Showing posts with label FabConEurope. Show all posts

Tuesday, 29 September 2026

OneLake is quietly becoming the trust layer for Data and AI

When Microsoft first introduced Fabric, much of the conversation focused on consolidation. The story was about bringing together data engineering, data warehousing, business intelligence and analytics into a single SaaS platform. OneLake played an important role within that narrative by providing a single storage layer that could be shared across workloads, reducing duplication and simplifying how data moved through the platform.



The announcements emerging from FabCon Barcelona suggest that Microsoft now has a much broader ambition for OneLake. While it remains the storage foundation for Fabric, many of the latest investments have surprisingly little to do with storage. Instead, they focus on governance, security, discovery, operational management and AI enablement. Viewed individually, features such as Governance Policies, Governance Insights, Domains, Secure and Catalog integration might appear to be incremental enhancements. Viewed collectively, they reveal a platform that is evolving beyond data storage and towards something much more strategic.

What stands out is that Microsoft appears to be tackling a different set of problems than those it faced when Fabric launched. Early adopters needed a platform capable of building data products and analytical solutions. Today's enterprise customers increasingly need a platform capable of operating hundreds or thousands of those assets while maintaining visibility, consistency and trust. The challenge is no longer simply creating a lakehouse, warehouse or report. The challenge is understanding who owns it, whether it can be trusted, how it should be governed and whether it can safely be used by both people and AI systems.

OneLake the operational centre of Fabric

One of the clearest signals from the latest announcements is the growing importance of the OneLake Catalog. Historically, data catalogues have been associated with discovery. They help users search for data, understand metadata and locate information assets across an organisation. What Microsoft is building increasingly goes beyond those traditional expectations.

The introduction of governance dashboards, governance policies, security administration and domain management within the catalog points towards a broader role. These capabilities are not helping users store data. They are helping organisations manage Fabric itself. The Govern experience provides visibility across the estate, Governance Policies introduce mechanisms for maintaining standards at scale, Domains establish accountability structures and the Secure experience provides oversight of access and permissions. These are operational capabilities rather than storage capabilities.

For organisations expanding their use of Fabric, this distinction matters. The technical challenge of creating analytical workloads is often relatively straightforward. The more difficult challenge is ensuring those workloads remain manageable as adoption grows. Platform teams need visibility. Governance teams need confidence. Administrators need control. The latest announcements suggest that Microsoft increasingly sees OneLake as the place where these concerns are brought together. This is perhaps one of the most significant shifts occurring within Fabric today. OneLake is no longer just where data resides. It is becoming the place where organisations understand and manage their data estate.

OneLake is becoming the discovery layer for trusted data

A second theme running through the announcements is trust. Most organisations do not have a shortage of data. If anything, they have the opposite problem. As data estates grow, users find it increasingly difficult to determine which assets are authoritative, which can be reused and which should inform business decisions. The challenge shifts from finding data to finding the right data.

Many of the catalog enhancements announced at FabCon can be viewed through this lens. Governance recommendations, endorsements, improved discovery experiences and governance insights all contribute towards helping users identify data that is trustworthy and reusable. Although these features appear distinct on the surface, they are addressing a common problem that affects almost every organisation pursuing data-driven transformation.

Trust has always been one of the more difficult aspects of governance to operationalise. Policies and standards can be documented, but users ultimately make decisions based on confidence. If they cannot easily determine whether data is owned, maintained, endorsed and understood, they will either avoid using it altogether or create their own alternatives. Neither outcome is desirable.

What makes the OneLake Catalog increasingly interesting is that it appears to be becoming the mechanism through which trust can be communicated. Ownership, endorsement, governance status, recommendations and business context are gradually being brought together into a single experience. Rather than simply helping users locate assets, the catalog is beginning to help users determine whether those assets should be used in the first place.

Consider OneLake as the Foundation for AI

The announcement that I find most revealing is not Governance Policies or the Govern experience. It is the decision to bring catalog discovery directly into Excel and Microsoft Foundry. At first glance this may appear to be a relatively modest enhancement. After all, users can already discover assets within Fabric itself. However, the significance becomes clearer when viewed from the perspective of adoption. Traditional data catalogues behave as destinations. Users must know they exist, remember to visit them and actively search for information. Microsoft appears to be pursuing a different approach. Instead of asking users to come to the catalog, the catalog is being embedded into the places where they already work.

For business users, that means trusted data can be discovered directly from Excel. For AI developers, it means information about governed organisational data can be surfaced within Foundry. In both cases, governance and discovery become part of the workflow rather than a separate activity.

This development is particularly important because AI systems face many of the same challenges as human users. An agent needs trusted information, ownership information, context and some means of understanding whether a dataset is authoritative or whether a conflicting source exists elsewhere. The role of the catalog therefore becomes increasingly important as organisations move beyond analytics and into AI-driven solutions.

Seen through this lens, the recent announcements are not simply governance enhancements. They are part of a wider effort to ensure that data can be safely discovered, understood and consumed by both people and AI systems.

The Bigger Picture

The easiest way to interpret the latest Fabric announcements is as a collection of new governance features. That explanation is not wrong, but it feels incomplete. A more interesting interpretation is that Microsoft is redefining the role of OneLake within the architecture. Governance, security, discovery and AI enablement are all being drawn closer to the storage layer and increasingly delivered through common experiences. The result is that OneLake is starting to look less like a data lake and more like the operational and trust layer that sits beneath the wider Fabric platform.

Whether this vision succeeds remains to be seen. However, the direction of travel is becoming increasingly clear. As organisations invest more heavily in Fabric, Microsoft's focus is shifting from helping customers build solutions to helping them govern, manage and trust those solutions at scale. This may not generate the same excitement as the latest AI capability, but it is precisely the sort of foundational investment that determines whether enterprise adoption can be sustained over the long term.

Reading

Build, deploy, and govern Microsoft Fabric at scale

FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and agents

Bringing governed analytics into the flow of work: Fabric Analytics at FabCon Europe 2026

What’s new across Microsoft SQL at SQLCon/FabCon Europe 2026

Microsoft Fabric at Scale: Technical Announcements for Administrators, Engineers and Platform Teams

It is the European Microsoft Fabric + SQL Community ConThe ference 28 Sep – 01 Oct 2026 in Barcelona. There are several Fabric specific announcements .  Arun Ulag’s blog FabCon and SQLCon 2026 in Barcelona: Building the data foundation for Microsoft Copilot and Agents contains a full list of announcements.




At FabCon, Microsoft's Build, Deploy and Govern Microsoft Fabric at Scale session focused less on end-user analytics and more on the practical realities of operating Fabric as an enterprise platform. The announcements were aimed squarely at the people responsible for platform engineering, DevOps, governance, security, and operational management. According to Microsoft, the themes of the session were CI/CD, agentic development, cost and resource management, observability, security, and governance, all designed to help organizations operate Fabric estates at scale without sacrificing control or compliance. Rather than discussing data products or business facing features, these announcements reflect Fabric's continued evolution into a mature enterprise platform that can support large-scale deployment, operational oversight, and governed self-service.

CI/CD Becomes a First-Class Experience. 

One of the strongest themes was Microsoft's continued investment in DevOps capabilities for Fabric.  Early Fabric adopters often found themselves balancing rapid innovation with the need for release controls, environment promotion, and deployment consistency. As Fabric workloads expanded beyond Power BI into data engineering, data science, real-time intelligence and databases, these requirements became significantly more important.

Microsoft's latest investments continue to strengthen CI/CD experiences, making it easier to treat Fabric assets as deployable platform components rather than isolated artefacts. This aligns Fabric more closely with modern software engineering practices where changes are versioned, tested and promoted through controlled deployment pipelines. For enterprise platform teams, this is particularly important because governance becomes easier when deployment processes are automated and repeatable. Manual configuration drift remains one of the most common causes of operational complexity in large data estates.

Why it matters:

Increased deployment consistency
Reduced risk of environment drift
Better support for enterprise release management
Stronger alignment between data engineering and software engineering teams

Agentic Development Arrives in Fabric

Another major direction highlighted by Microsoft was agentic development. The Fabric platform is increasingly embracing AI-assisted development experiences that help users build solutions faster while still operating within governed environments. Rather than simply generating code snippets, Microsoft's vision is for AI-powered assistants that can understand platform context, automate repetitive development tasks, and improve developer productivity.

This reflects a wider industry trend where platform engineering teams are looking to accelerate delivery without continually increasing headcount. AI tooling is becoming part of the development lifecycle itself, reducing administrative burden while helping teams navigate increasingly complex environments. The challenge for many organisations will not be whether they use AI-assisted development, but how they ensure these capabilities operate within security, compliance, and governance controls.

Why it matters:

Faster development cycles
Reduced platform administration overhead
Increased developer productivity
Better scaling of engineering teams

Observability Becomes a Critical Capability

As Fabric estates grow, monitoring becomes significantly more important. Microsoft highlighted continued investment in observability and operational visibility, helping administrators understand how capacities are performing, where resources are being consumed, and how workloads are behaving.

This is particularly relevant because many organizations have now moved beyond small pilot deployments. They are running production workloads, serving large user communities and supporting critical business processes. At that scale, platform teams need visibility into reliability, performance bottlenecks and resource consumption trends. Recent roadmap announcements further reinforce this direction with enhanced capacity monitoring, utilization insights, throttling visibility, scaling controls and capacity health indicators becoming part of the platform experience.

Why it matters:

Faster issue identification
Improved platform reliability
Better capacity planning
Reduced operational risk

Improved Cost and Capacity Management

Managing Fabric costs has become a major focus for many organisations. As adoption increases, conversations often move beyond functionality and towards sustainability. Platform owners need confidence that they can scale usage without unexpected capacity challenges or performance degradation.

Microsoft's announcements around cost and resource management indicate continued investment in helping organisations understand and control consumption. This includes better visibility into resource utilisation and more sophisticated capacity management capabilities. This is particularly important for organisations adopting a hub-and-spoke operating model where multiple business units share platform resources.

Why it matters:

Greater financial transparency
More predictable platform operations
Better chargeback and showback models
Improved utilisation of Fabric capacities

Security and Governance Continue to Mature

Security and governance were also central themes throughout the announcements. Microsoft continues to position Fabric as an enterprise-ready platform that integrates governance, compliance, auditing, security controls and administrative oversight directly into the platform. Microsoft's governance documentation highlights capabilities such as audit, workspace governance, tenant controls, monitoring, lineage, information protection and broader integration with Microsoft Purview.

The significance of these announcements is not simply that governance features exist. It is that governance is increasingly becoming operationalised as part of day-to-day platform management rather than remaining a separate compliance exercise. For platform teams, this means security and governance controls can be embedded alongside deployment, monitoring and operational management processes.

Why it matters:

Stronger enterprise compliance
Improved security oversight
Reduced governance debt
Better support for AI-ready data estates

The Bigger Picture

Taken individually, many of these announcements appear incremental. Together, however, they reveal Microsoft's strategic direction for Fabric.  Fabric is no longer just a collection of analytics workloads. Microsoft is steadily building the supporting capabilities required to operate a large-scale enterprise data platform. The focus on CI/CD, observability, cost management, security, governance and AI-assisted development demonstrates a platform increasingly designed for enterprise-scale operation rather than departmental adoption alone.  For administrators, platform engineers and governance teams, this is arguably the most important message from the session. The newest capabilities are less about creating another report or building another pipeline. They are about enabling organisations to run Fabric confidently at scale while maintaining reliability, security and control. In many ways, these were the announcements that move Fabric from a powerful analytics platform towards a mature operating platform for enterprise data, AI and analytics.

Further Reading

Wednesday, 25 September 2024

European Microsoft Fabric Community Conference Keynote

The Microsoft Fabric Community Conference is taking place this week in Stockholm, Sweden. There are 3300 attendees which is amazing.  The Keynote was entitled Microsoft Fabric Vision & Roadmap – Analytics in the Era of AI.  The keynote speakers; Arun Ulag, Kim Manis, Amir Netz, Marco Casalaina, Wangui McKelvey and Rudra Mitra from Microsoft.

The announcements blog contains a vast number of innovative enhancements and capability updates. The blog is: European Fabric Community Conference 2024: Building an AI-powered data platform

Fabric is about providing a unified data platform for AI transformation with these 3 areas

  • AI-powered development: Fabric can give teams the AI-powered tools needed for any data project in a pre-integrated and optimized SaaS environment.
  • An AI-powered data estate: Fabric can help you access your entire multi-cloud data estate from a single, open data lake, work from the same copy of data across analytics engines, and use that data to power AI innovation 
  • AI-powered insights: Fabric can empower everyone to better understand their data with AI-powered visuals and Q&A experiences embedded in the Microsoft 365 apps they use every day. 

Microsoft Fabric is shown in the AI Copilot stack.

There is a new exam available to enhance skilling in data engineering DP-700 This Certification measures Fabric data engineer skills:

  • Implementing and managing an analytics solution
  • Ingesting and transforming data
  • Monitoring and optimizing an analytics solution

Bringing all the skills together for data engineering, data science, and data analytics. You can read more here

 There  are numerous technical announcements




You can see the list of GA features shipped for network security, data security and governance. There is an End-to-End Security Whitepaper to the read aka.ms/FabricSecurityWhitepaper 

The public preview of the new copy job item provides
  • Built-in support for copying data between workspaces
  • Streamline copying data into and across OneLake
  • Setup automatic incremental copy

To help with deployment, the public preview was announced of the Terraform Provider for Microsoft Fabric. This will help empower deployment, management, and governance. This Microsoft Fabric Infrastructure as Code (IaC) new Terraform Provider lets you use the same tools and processes already employed for Microsoft Azure and other cloud resources.

Access Azure Databricks Unity Catalog tables directly from Fabric via the new Mirrored Azure Databricks Catalog feature, now in Public Preview https://blog.fabric.microsoft.com/en-US/blog/databricks-unity-catalog-tables-available-in-microsoft-fabric/




All the in dept announcements are on the Fabric Blog. 

The unified data culture was mentioned between Fabric and Office connected through Power BI. 


There are lots more releases out there. This is such an exciting time in the Data and AI space. The exciting announcement is that the conference will be back next year. 


Good Morning European Microsoft Fabric Community Conference

 Good Morning for day 2 of the European Microsoft Fabric Community Conference