Big Data London emerging insights

It was great to be at BigDataLDN this week and see the changing landscape. 40% of all sessions had AI in the title. This is a significant shift compared to pre 2023 programmes, where AI titled sessions were typically less than 15%.

The 2026 landscape highlighted a number of shifts.

AI is no longer a track, it is the spine of the conference embedded across every discipline.

Governance and trust are now centre stage. Multiple sessions explicitly focused on trusted AI foundations, AI governance maturity, scaling AI safely, data governance as the prerequisite for AI and observability for AI systems.

Agentic AI is emerging as a major theme. This is the first year agentic AI had a dedicated theatre signalling a shift from the previous experimentation phase to operationalisation.

AI plus organisational transformation is the new battleground asking questions like how do we scale AI safely,  modernise legacy environments,  demonstrate measurable business value, and improve governance without slowing innovation.  This is a change from asking which model to use towards to how do we run an AI enabled organisation. 

Data governance is finally being treated as strategic and it was the strongest governance representation BigDataLDN has ever had.

AI for societal impact is gaining traction with talks now covering climate change and humanitarian impact broadening AI use beyond commercial use cases.

The keynotes were AI centric which created a conference theme of AI first.

I think the BigDataLDN conference has quietly repositioned itself an AI governance, AI strategy, AI engineering and AI transformation conference. 

AI governance seems like the new cloud migration in that it requires a change of mindset. The agenda showed a pattern where

• AI adoption is assumed. 
• AI scaling is the challenge. 
• Governance is the bottleneck. 
• Data foundations are the dependency. 
• Operating models are the differentiator

The conference was packed with people and exhibitors and many people spent their time queueing to get into rooms to get to listen to talks.

 

Responsible AI in 2026: Governance Moves from Principle to Practice

Microsoft's latest article, Responsible AI in 2026: How We Are Adapting for What's Ahead, highlights something many of us working in governance have been seeing for some time: AI governance is no longer a future concern. It is becoming an operational necessity. 

As AI capabilities continue to accelerate, particularly with the rise of agentic AI, the governance challenge is changing. Traditional governance approaches were largely focused on data, systems, and applications. Increasingly, organisations must also govern autonomous actions, agent interactions, tool permissions, and dynamic decision-making processes. Microsoft describes this as a move towards more adaptive governance, where controls evolve alongside the capabilities and risks of AI systems. 

What I found most interesting is that the article places relatively little emphasis on the models themselves and much more emphasis on governance, risk management, monitoring, and assurance. Microsoft explicitly states that model capability alone will not determine AI's impact. Success will depend on whether organisations can govern AI with the rigour and adaptability needed to earn trust. 

This mirrors a trend I am seeing across the market. Many organisations are still focused on AI adoption, Copilot deployments, and proof-of-concepts. However, the harder question is emerging quickly: how do we maintain visibility, accountability, and control once AI becomes embedded in day-to-day operations?

The answer is unlikely to be found in technology alone. Microsoft's report discusses governance frameworks, risk management processes, evaluation capabilities, training, standards, and industry collaboration. These are all governance disciplines rather than purely technical controls. 

For data governance professionals, this should sound familiar. The foundations that organisations have spent years developing around ownership, accountability, quality, security, and compliance are becoming even more important in an AI-enabled world. AI governance is not replacing data governance. It is extending it.

Perhaps the most significant message from the article is that responsible AI cannot be treated as a static policy document. Microsoft describes governance as a continuous lifecycle activity that must evolve as systems learn, interact, and operate in increasingly complex environments.  That is a valuable lesson for every organisation currently exploring AI. The conversation is no longer about whether governance matters. The conversation is about whether governance can keep pace with AI's rapid evolution.

As Microsoft's latest transparency report demonstrates, the organisations most likely to succeed with AI will not simply be those with access to the best technology. They will be the organisations that can combine innovation with trust, control, and effective governance.

References

https://blogs.microsoft.com/on-the-issues/2026/09/01/responsible-ai-in-2026-how-we-are-adapting-for-whats-ahead/

Governance must keep pace with AI and be embedded in every stage

Over the last few weeks, the conversation around artificial intelligence has taken an increasingly dramatic turn. Following Dario Amodei's essay, We Must Pace the Frontier, and widespread media coverage of warnings from researchers and technology leaders, discussions about AI have become dominated by questions of existential risk, cyber warfare, loss of control and the possibility that advanced systems could outpace human oversight. Amodei's central argument is that the rate of AI capability development may be accelerating faster than our ability to understand, govern and safely manage those capabilities, creating a situation where precaution needs to catch up with progress.

These AI fears made me think of pushing beyond design limits where Donald Campbell’s final attempt in 1967 on Coniston Water pushed Bluebird K7 past 300 mph far beyond its original design rating of 250 mph. This pushing technological boundaries to shatter another world record, demonstrated that accelerating past design limits without evolving the safety framework exposes fatal vulnerabilities.

While these concerns deserve serious consideration, I have been struck by how many of the proposed solutions focus on slowing AI itself. The assumption seems to be that if technology advances too quickly, the safest response is to reduce the speed of innovation until regulators, policymakers and society have time to react. However, I am not convinced that slowing AI addresses the underlying issue. The problem is not that artificial intelligence exists or that organisations are finding new ways to apply it. The problem is that governance continues to lag behind technological change, despite decades of evidence showing that this always creates unnecessary risk.

Every major technological shift follows a remarkably similar pattern. Organisations become excited by new capabilities, investment accelerates, adoption grows rapidly and governance is treated as something that can be addressed later. Eventually the consequences of that approach become visible, whether through security incidents, compliance failures, poorly understood risks or loss of trust. The discussion then turns towards regulation, controls and accountability. What is often forgotten is that governance could have been embedded from the beginning.

The current debate around AI increasingly focuses on the possibility that advanced systems may one day become difficult to control. Yet many organisations are already struggling with far more immediate challenges. They do not know who owns critical datasets. They cannot consistently identify authoritative information. They have limited visibility of the quality of the data entering analytical platforms. They have duplicated reports, conflicting definitions and inconsistent security controls. These are not theoretical future concerns. They are today's governance problems, and AI simply amplifies them.

This is one of the reasons I find the current distinction between data governance and AI governance increasingly key. AI governance is undoubtedly important, particularly as organisations begin deploying copilots, autonomous agents and decision-support systems. However, the majority of the risks associated with AI are ultimately rooted in issues that data governance has been trying to solve for years. Questions about ownership, accountability, transparency, lineage, quality, security and trust do not suddenly appear because an organisation deploys an AI model. Those questions already existed. AI merely exposes them more quickly and at greater scale.

Consider the current wave of Microsoft Copilot deployments taking place across both public and private sector organisations. There is understandable excitement about productivity gains and new ways of working, but Copilot does not create knowledge. It surfaces what already exists inside the organisation. The challenge is the state of the information environment that AI is consuming.

What concerns me most is that governance is still frequently discussed as if it were a specialist discipline owned by a single team. The reality is that the next generation of technology will make that approach increasingly difficult to sustain. As organisations move towards more autonomous forms of AI, governance decisions will need to be incorporated directly into project delivery, operational processes, architecture reviews, software development lifecycles and technology investment decisions. It will not be sufficient to maintain a separate governance workstream running alongside change initiatives. Governance will need to become a fundamental characteristic of how change is delivered.

This becomes particularly important when considering the rise of agentic AI. Much of today's governance discussion focuses on whether an AI model is accurate, fair or explainable. Those questions remain important, but autonomous systems introduce an entirely new set of concerns. Organisations will need to understand who is accountable for actions taken by an agent, what permissions it possesses, how its behaviour is monitored, when human intervention is required and how decisions are audited. These challenges cannot be resolved through model governance alone. They require broader governance frameworks that connect business ownership, risk management, security and information management.

For this reason, I believe the debate about whether we should slow AI down is asking the wrong question. The more important question is whether governance can evolve quickly enough to keep pace with innovation. History suggests that organisations are capable of managing significant technological change when appropriate governance structures are embedded from the outset. We have done this with financial controls, health and safety, privacy, cyber security and regulatory compliance. None of these disciplines emerged because organisations stopped innovating. They emerged because innovation required new forms of oversight and accountability.

If the concerns raised by Dario Amodei prove justified, then the answer is unlikely to be found solely through reducing the pace of technological development. The more sustainable response is to ensure that governance develops at the same speed as the technologies it is intended to support. Data governance, AI governance, security governance and risk management should not be viewed as separate initiatives competing with innovation. They should be recognised as the mechanisms that make innovation sustainable.

The future of AI will undoubtedly introduce challenges that we have not yet anticipated. However, organisations do not need to wait for hypothetical existential threats before they strengthen governance. The foundations are already well understood. Ownership, accountability, transparency, stewardship, good data quality, security and trust remain as relevant today as they were before the first large language model entered the public consciousness. The difference is that AI has transformed these disciplines from desirable good practice into essential business capabilities.

The organisations that succeed over the next decade will not necessarily be those that adopt AI first or deploy the greatest number of models. They will be the organisations that recognise governance as an enabler of innovation rather than a constraint upon it. In a world where AI is becoming embedded into every platform, every process and every decision, governance must become equally pervasive. The challenge is not slowing AI down. The challenge is ensuring that governance finally catches up.

References

We Must Pace the Frontier https://darioamodei.com/post/we-must-pace-the-frontier

The Guardian — “‘We must slow the pace’: CEO of Anthropic calls for an AI slowdown

https://www.theguardian.com/technology/2026/sep/12/we-must-slow-the-pace-ceo-of-anthropic-calls-for-an-ai-slowdown

TechRepublic — “Altman, Musk Back Amodei’s AI Warning: The Frontier May Be Moving Too Fast” https://www.techrepublic.com/article/news-amodei-altman-musk-slow-frontier-ai/

BBC Why are there concerns AI could threaten humanity, and how real are they? https://www.bbc.co.uk/news/articles/c790xvnzgnno

BBC AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC https://www.bbc.co.uk/news/articles/cqgk5e2j0gg8o

BBC Anthropic researcher believes more than 10% chance AI 'could kill all humans' https://www.bbc.co.uk/news/articles/ckgwy1k42w4o

Microsoft Fabric Ontology: The missing layer between Data Governance and AI

The data industry has spent the last twenty years focused on one primary challenge: connecting data. We built and created increasingly sophisticated ways of moving information between systems and making it available for analytics. Many organisations today still struggle with a much simpler problem and that is clarity on terminology. Different department terms often mean different things when they use the same business terms.

A customer means one thing in CRM, another in finance and something slightly different again in marketing. Product definitions vary between commercial teams and operational systems. Employee records, supplier information, and assets frequently exist across multiple applications, each with its own interpretation and business rules.

Humans have become relatively adept at navigating these differences because they understand organisational context. AI does not understand this. As organisations increasingly adopt Copilot, AI agents and intelligent business applications, the challenge is no longer giving AI access to data but giving AI the meaning of terms.

This is where Microsoft Fabric Ontology, currently in preview as part of Fabric IQ, becomes particularly interesting. Microsoft describes Ontology as a machine understandable representation of enterprise vocabulary that defines business concepts through entity types, properties and relationships, creating a shared business context layer that can be used across teams, applications and AI agents. 

What makes this significant is not the technology itself but how important the need is for AI to have business understanding, just as much as it requires data access.

The problem we have been trying to solve for years

Anyone who has worked in data governance will recognise this challenge immediately. We have built business glossaries, data dictionaries, conceptual models and reference architectures in an attempt to create consistency across the organisation. Governance programmes have invested significant effort defining critical data elements, agreeing business terminology and establishing ownership for key information assets. The difficulty has always been turning those definitions into something operational. Many governance initiatives successfully define what a customer is, but those definitions often remain trapped in documents, spreadsheets or governance tools that sit separate from the systems actually using the data. The glossary becomes a reference point for people rather than an active component of the architecture. As a result, governance knowledge frequently exists in one location whilst operational data exists somewhere else. Both are valuable, but the connection between them is often weak.

Microsoft's vision for Ontology appears to be closing that gap. Rather than maintaining business definitions separately from data, Ontology allows organisations to define core business concepts and then bind those concepts directly to data residing within OneLake, Power BI semantic models, lakehouses and other Fabric data sources. The result is that the business definition and the physical data become connected through a common semantic layer. From a governance perspective, the business glossary stops being passive documentation and becomes a part of how information is understood and consumed throughout the platform.

Why AI changes everything

Inconsistencies in business terminology are often frustrating but manageable. Analysts learn the system nuances and data engineers write transformation logic to reconcile differences. Often reporting teams spend their time explaining why the numbers vary between departments.

AI fundamentally changes the scale of the problem. When an AI agent is asked a question such as Which customers are most at risk of churn? it needs more than access to customer records. It needs to understand what a customer is, which systems contain authoritative information, how related concepts connect to one another and which business rules should be applied during analysis. Without that context, even a highly capable model can produce inconsistent or misleading outcomes. 

Microsoft specifically highlights Ontology as a shared business context layer that can be consumed by Fabric agents and AI-driven workflows to support reasoning and actions across domains. Rather than asking questions against individual tables, users and AI agents can query business concepts that already carry organisational meaning.

As organisations move beyond simple AI assistants and towards agentic architectures where AI systems are expected to make decisions, execute processes and reason across multiple business domains having this tool is important. The quality of decisions will depend heavily on the quality of the organisational context provided to them.

More than another Semantic Model

Ontology is not simply another version of a semantic model. Semantic models primarily exist to simplify analytics and reporting. They provide a business friendly representation of data designed to support measures, calculations and reporting experiences.

Ontology aims to tackle a much broader challenge. It introduces concepts such as entity types, properties and relationships that represent how the organisation understands the world. Customer, Supplier, Product, Contract and Asset become business entities that exist independently of any particular source system. Relationships become explicit rather than buried inside data models and joins.

Microsoft also introduces a graph representation that allows relationships between entities to be stored and queried directly. In practical terms, this means understanding not just what something is, but how it connects to everything around it. Customers place orders. Suppliers provide products. Employees manage projects. Assets support services. These connections become part of the semantic model itself rather than logic recreated repeatedly by individual development teams. This kind of contextual understanding  for AI is enormously valuable because reasoning is often driven by relationships as much as by data values.

Is Ontology replacing Microsoft Purview?

One of the most common questions I have seen since the announcement is whether Ontology makes Microsoft Purview less relevant. Ontology and Purview address different layers of the same challenge.

Purview focuses primarily on understanding, governing and protecting information assets. It discovers data, provides lineage, manages classifications, supports compliance activities and enables organisations to establish trust in their information landscape.

Ontology focuses on meaning. 

Where Purview helps answer questions such as Where is this data?, Who owns it?, How sensitive is it? and Where did it come from?, Ontology helps answer questions such as What does this represent?, How does it relate to other business concepts? and How should AI reason about it?

The two capabilities compared.

Capability	Microsoft Fabric Ontology	Microsoft Purview
Primary objective	Create shared business meaning	Govern and manage enterprise data
Key focus	Business concepts and relationships	Data assets and metadata
Business glossary	Operational semantic layer	Governance glossary and terminology
AI support	Grounding and business reasoning	Trusted metadata and governance controls
Relationships	Business relationships between entities	Data lineage and technical relationships
Graph capabilities	Native graph-based business context	Metadata and lineage visualisation
Data discovery	Bound Fabric data sources	Enterprise-wide discovery
Classification	Limited focus	Core capability
Compliance	Not a primary objective	Core governance capability
Security and risk	Relies on platform controls	Governance, risk and compliance controls
Typical audience	AI teams, business architects, domain experts	Data governance, security and compliance teams

What this means for the Future of Governance

For me, the most significant aspect of Ontology is what it says about the future direction of governance. Historically, data governance was largely created for people. Policies were written for humans, glossaries were maintained for humans. with data standards interpreted by humans. Increasingly, we need governance artefacts that machines can understand directly. AI agents, Copilots and autonomous systems cannot read a governance policy and infer organisational meaning in the way people do. They need structured, machine readable context. They need agreed definitions and relationships. Also business vocabulary they can reason over consistently is required.

Ontology appears to be Microsoft's recognition that the next generation of governance must serve both humans and machines. As AI becomes embedded within business operations, organisations will increasingly discover that trusted data is only part of the equation. Equally important is ensuring that AI understands what that data actually means. Data governance professionals have argued for years that data without context has limited value. In the era of enterprise AI, that statement feels more true than ever. Trusted AI requires trusted data, but it also requires trusted meaning. Fabric Ontology is a significant step towards delivering that meaning at scale.

Reference

What is ontology (preview)?

Microsoft Tools for Making Data AI-Ready

Building AI for Human Flourishing: Inside Microsoft’s Humanist AI Code of Conduct

As AI accelerates toward super intelligent capabilities, Microsoft has taken a bold and transparent step: publishing the draft Humanist AI Code of Conduct today, a governing framework designed to ensure future AI systems remain safe, aligned, and firmly under human control.

At its core, the Code of Conduct is built on a simple premise people matter more than AI. Technology should amplify human wellbeing, expand opportunity, and strengthen human judgment rather than replace it. Microsoft’s Humanist AI approach rejects the pursuit of AI personhood or consciousness like behaviour, instead focusing on systems that are powerful, purposeful, and safely constrained.

The document outlines several pillars:

• Human Control & Safety — AI must remain subordinate to humanity, never resisting shutdown, redirection, or oversight. Safety constraints cannot be overridden by users or operators.  
• Clear Boundaries — Models will not assist with weapons, mass harm, offensive cyberattacks, manipulation at scale, or any content that violates human dignity or child safety.  
• Human Flourishing — AI should enhance learning, creativity, collaboration, and wellbeing, helping people make better decisions without replacing personal growth or human relationships.  
• Pluralism & Inclusion — AI should support diverse cultures, values, and perspectives while upholding universal human rights and avoiding harmful bias.  
• Transparency & Public Input — Microsoft is inviting global feedback to refine the Code before it becomes the governing blueprint for model development in 2027 and beyond.

This draft marks a significant moment: a major technology company openly sharing how it intends to build and govern superintelligent systems. It signals a future where AI is not an autonomous force but a carefully designed partner, shaped by human values, guided by public dialogue, and constrained by safety-first engineering.

References 

Humanist AI Code of Conduct

https://microsoft.ai/code-of-conduct/

Lineage aware AI in Microsoft Fabric: A New Era of Intelligent Data Context

Microsoft Fabric has introduced a preview feature that quietly unlocks something powerful: programmable lineage. With the new Item Relations API, developers and data teams can finally access the same dependency information shown in the Fabric lineage view but now through REST endpoints that automation and AI can understand.

Lineage has always been essential for responsible data engineering. It tells you how items connect, what depends on what, and where changes might cause disruption. Until now, that insight lived mostly inside the Fabric UI. The new API changes that completely.

What the API enables

The Item Relations API exposes upstream and downstream relationships for any Fabric item. It also returns typed edges such as shortcuts, associations, orchestration links, and push‑data flows giving a structured map of how the data estate fits together.

This means the tools can now:

- Perform impact checks before modifying or deleting assets  

- Generate automatic documentation that stays current  

- Build governance dashboards driven by real dependency graphs  

- Give AI agents the context they need to reason about data safely  

Instead of guessing, the automation can now see the actual lineage.

Why this matters for AI

AI systems are only as smart as the context they receive. By exposing lineage through an API, Fabric allows AI to understand not just the content of a dataset, but its role in the wider environment. That’s a foundational step toward safer, more reliable AI‑driven automation.

A strategic preview

Although still in preview, the Item Relations API signals a shift: Fabric is turning metadata into an actionable surface for engineering, governance, and AI. It’s a small feature with big implications for anyone building responsibly in a complex data landscape.

Reference

Lineage-aware AI with the Fabric item relations API (Preview)

https://community.fabric.microsoft.com/blog/fbc_fabricupdatesblogs/lineage-aware-ai-with-the-fabric-item-relations-api-preview/5366064

The Hierarchy of AI Oversight

Organisations frequently treat Responsible AI, AI Governance, and Data Governance as synonymous concepts. In practice, they represent three distinct, interdependent structural tiers. Treating them as interchangeable obscures how AI systems are built, verified, and operationalised within an enterprise.

A useful way to conceptualise this structure is through a three-part stack:

•  Responsible AI defines organizational intent and boundaries.
•  AI Governance establishes operational execution and control mechanisms.
•  Data Governance manages the underlying assets and pipeline inputs.

When any single tier is neglected, the entire oversight framework becomes ineffective.

Responsible AI: Establishing Strategic Intent

Responsible AI sits at the top of the stack as an explicit declaration of intent. It articulates an organisation's risk tolerance, core values, and societal commitments regarding automated systems.

This layer does not detail specific technical configurations or workflow steps. Instead, it defines the overarching ethical perimeter, addressing core themes such as non-discrimination, explainability, safety, and accountability.

Key questions addressed at this layer include:

•  What operational boundaries define acceptable versus unacceptable AI deployments?
•  What specific harms must system designs actively prevent?
•  What baseline commitments are required for external stakeholders and regulatory bodies?

While Responsible AI acts as the strategic compass, policy statements alone do not alter system behavior. Without operational enforcement, policy declarations remain purely symbolic. Operationalising these policies requires the secondary layer: AI Governance.

AI Governance: Implementing Operational Control

AI Governance provides the operational apparatus required to enforce Responsible AI policies. It consists of the decision rights, verification protocols, audit trails, and risk taxonomies that manage an AI model across its complete lifecycle.

This tier shifts abstract commitments into concrete engineering and management workflows. It covers model validation standards, change management, automated drift detection, and post-deployment monitoring. Systems like the GRAICE framework operate within this domain to standardise evaluation criteria.

Key questions addressed at this layer include:

• Which roles hold approval authority at distinct stages of model development?
• What quantitative evidence is required prior to production deployment?
• How are performance degradation, bias drift, and unexpected edge cases detected and remediated?
• What specific conditions trigger a mandatory model recall or pause?

AI Governance ensures that models operate within defined parameters over time. However, governance controls cannot ensure model integrity if the underlying inputs are flawed. Control frameworks require verifiable data inputs, which depends entirely on the foundational layer.

Data Governance: Securing the System Inputs

Data Governance manages the quality, legal basis, security, and lineage of the data fed into machine learning pipelines. Because statistical models reflect the characteristics of their training data, AI performance is constrained by the quality of its underlying data architecture.

Without robust data management, model output becomes inherently unpredictable. Issues such as unverified data sources, unrecorded pipeline transformations, or demographic skew directly compromise model outputs regardless of how stringent the AI control checks are.

Key questions addressed at this layer include:

•  What is the precise lineage and chain of custody for training and validation datasets?
•  Do clear usage rights, legal bases, and consent frameworks exist for the ingested data?
•  Is the dataset representative, accurate, and properly versioned?
•  How are data access controls and privacy-preserving techniques maintained through the pipeline?

Strong Data Governance provides the verifiable evidence base that AI Governance relies on. Without it, validation processes lack technical substance

.

Structural Pitfalls of Top-Down Implementation

A common failure mode occurs when organisations implement oversight from the top down. Leadership teams often publish high-level ethical guidelines and establish oversight committees before building the necessary operational controls or securing data infrastructure.

This top-down approach creates several operational vulnerabilities:

•  Oversight committees evaluate systems without reliable technical lineage or performance data.
•  Data quality defects and unverified assumptions are identified late in production rather than during ingestion.
•  Ambiguity surrounds technical accountability when failures occur.
• Defining ethical principles without establishing underlying governance frameworks leads to superficial compliance—where policy commitments exist on paper but cannot be verified or enforced at the engineering level.

Building a Cohesive Oversight Framework

Establishing an effective oversight framework requires starting from foundational technical controls and building upwards:

• Establish Data Integrity: Secure data lineage, document legal rights, enforce validation checks, and maintain clear data stewardship across all pipelines.
• Deploy Control Architectures: Implement repeatable stage-gate approvals, continuous testing protocols, risk logging, and lifecycle monitoring.
• Align Operational Controls with Policy Boundaries: Connect technical metrics and threshold alerts directly to high-level organizational principles and regulatory requirements.

Aligning these three disciplines transforms AI oversight from a collection of isolated policies into an integrated operational capability.