Passionately curious about Data, Databases and Systems Complexity. Data is ubiquitous, the database universe is dichotomous (structured and unstructured), expanding and complex. Find my Database Research at SQLToolkit.co.uk . Microsoft Data Platform MVP

"The important thing is not to stop questioning. Curiosity has its own reason for existing" Einstein

Monday 4 December 2023

Data Toboggan Winter Edition 2024

We are pleased to announce that Data Toboggan will return for its 4th year on Saturday 3rd February 2024. Our 2 track virtual conference covering covering all aspects of Azure Synapse Analytics and Fabric including Dedicated SQL Pools, Serverless SQL Pools, Spark, Pipelines, Data Engineering, Power BI, Machine Learning, and much more!

Imaged created in https://designer.microsoft.com/

Saturday 2 December 2023

Live view in Microsoft Purview

Live view in Microsoft Purview is in preview. Live view is a place that users with data access permissions can find Azure resources and their data assets in the Microsoft Purview Data Catalog without set-up or scanning.  The metadata of the resources is immediately available when you access the free or enterprise versions of Microsoft Purview. 

There are some limitations for live view for Microsoft Fabric:

  • Personal workspaces aren't supported yet in live view.
  • Only Microsoft Fabric item level metadata are available in live view.
  • Power BI Datasets don’t support showing schema in live view if the item hasn’t been scanned.
  • Dataflow Gen2 (Preview) items are displayed as Dataflow items.

Microsoft Fabric items  that will be available in the Microsoft Purview Data Catalog using live view are:

Wednesday 29 November 2023

A Generative AI course for beginners

Microsoft Cloud Advocates have created a new set of training materials to enable the learning of the fundamentals of building Generative AI applications with a 12-lesson comprehensive course. Each lesson covers a key aspect of Generative AI principles and application development. Throughout this course, you will be building your own Generative AI start up so you can get an understanding of what it takes to launch your ideas. The agenda is below.

Lesson Link
Course Introduction - How to Take This Course
Concepts Taught

Tech setup and course structure
Learning Goal
Setting you up for success while learning in this course

Introduction to Generative AI and LLMsConcept: Generative AI and the current technology landscapeUnderstanding what Generative AI is and how Large Language Models (LLMs) work.

Exploring and comparing different LLMsConcept: Testing, iterating, and comparing different Large Language ModelsSelect the right model for your use case

Using Generative AI ResponsiblyConcept: Understanding the limitations of foundation models and the risks behind AILearn how to build Generative AI Applications responsibly

Understanding Prompt Engineering FundamentalsCode/Concept: Hands-on application of Prompt Engineering Best PracticesUnderstand prompt structure & usage

Creating Advanced PromptsCode/Concept: Extend your knowledge of prompt engineering by applying different techniques to your promptsApply prompt engineering techniques that improve the outcome of your prompts.

Building Text Generation ApplicationsCode: Build a text generation app using Azure OpenAIUnderstand how to efficiently use tokens and temperature to vary the model's output

Building Chat ApplicationsCode: Techniques for efficiently building and integrating chat applications.Identify key metrics and considerations to effectively monitor and maintain the quality of AI-powered chat applications

Building Search Apps Vector DatabasesCode: Semantic vs Keyword search. Learn about text embeddings and how they apply to searchCreate an application that uses Embeddings to search for data.

Building Image Generation ApplicationsCode: Image generation and why it's useful in building applicationsBuild an image generation application

Building Low Code AI ApplicationsLow Code: Introduction to Generative AI in Power PlatformBuild a Student Assignment Tracker App for our education startup with Low Code

Integrating External Applications with Function CallingCode: What is function calling and its use cases for applicationsSetup a function call to retrieve data from an external API

Designing UX for AI ApplicationsConcept: Designing AI Applications for Trust and TransparencyApply UX design principles when developing Generative AI Applications

Continue Your LearningLinks to continue your learning from each lesson!Mastering your Generative AI skills

Saturday 18 November 2023

Microsoft Fabric is Generally Available

I watched the session 

Make your data AI ready with Microsoft Fabric and Azure Databricks

Jessica Hawk, Amir Netz, Arun Ulagaratchagan, Patrick Baumgartner, Profile picture of Kristen Christensen, Justyna Lucznik

'Bring your data into the era of AI with Microsoft Fabric, a powerful all in one AI powered analytics solution for enterprises that covers everything from data movement to data science, real time analytics and business intelligence. Learn how Azure Databricks and Microsoft Fabric seamlessly work together to offer customers a modern, price performant analytics solution that helps teams turn data into a competitive advantage.'

The discussion centred around the general availability of Microsoft Fabric. Below are are few slide images from the presentation.

Here is some more information about the changes.

Prepare your data for AI innovation with Microsoft Fabric—now generally available

Microsoft Fabric, explained for existing Synapse users

Friday 17 November 2023

The new Microsoft Purview portal (preview)

The new Microsoft Purview portal helps you discover and access data security, data governance, and risk and compliance solutions for all your data in your data estate. This unified experience streamlines navigation for all Purview solutions and provides a single-entry point for settings, search, and roles and permissions management.  The risk and compliance solutions in the portal currently opens these solutions in the classic Microsoft Purview compliance portal . The portal inclusions:

New Microsoft Purview portal (preview) can be found at: https://purview.microsoft.com

The solution cards on the portal home page provide quick access to open the Microsoft Purview solutions where access has been granted. It is great to see the Purview portals coming together under one roof as this will make navigation easier. 

Thursday 16 November 2023

Microsoft Ignite 2023

An exciting introduction to Microsoft Ignite November 15 - 17, 2023. Two themes that I am excited about are the AI and Microsoft Fabric capability. 

Microsoft Ignite 2023: AI transformation and the technology driving change

The full list of changes at Microsoft Ignite are listed in the Microsoft Ignite Book of News.

In the main keynote Satya announced over 100 changes. A few of the announcements discussed in the keynote are mentioned here:

Azure Maia - The Microsoft Cloud accelerator optimized for LLM training and inference

Azure Open AI Service  updates

  • GPT-4 Turbo - Preview
  • GPT-4 Turbo with Vision - Preview
  • DALL-E 3 - Preview
  • Fine Tuning - Preview

Model catalog - New

  • Stable Diffusion
  • Code Llama
  • Mistral 7B
  • NVIDA Nemotron-3 8B

Models as a Service - announcing

  • Ready to use APIs
  • Hosted finetuning
  • Integrated with leading LLM tools

Phi-2 - Announcing

  • Built by Microsoft Research
  • 2.7 billion parameters
  • Common sense, language understanding, and logical reasoning

The Phi-2 model is changing the game in artificial intelligence (AI) and it is designed to make machines think more like humans and do it safely.

Azure AI Studio - Preview
  • Unified platform
  • Hybrid and sematic search
  • Full development lifecycle
  • Extends to Windows
  • Safe and Responsible AI
The Copilot stack is evolving with greater functionality.

Microsoft Fabric is now generally available, a data platform for the era of AI.

Azure AI Search - Deliver accurate, hyper-personalized responses in your Gen AI applications

There was also announcement of Mesh on Microsoft Teams being generally available. 

  • Copilot and avatar
  • Immersive spaces
  • No-code editor
  • Copilot and 3D space

Then finally something different Azure Quantum Elements - purpose built to accelerate scientific discovery.

PASS Community Summit 2023 Keynote 1

The PASS Data Community Summit 2023 global data conference is happening in Seattle between 14-17 November 14-17 2023.  

The first keynote 'Limitless growth, limitless opportunities: Data and innovation in an AI world' was given by  Shireesh Thota. He started talking about new technologies and how they offer limitless potential with AI, Generative AI and Real-time capabilities.  The limitiless growth with Azure and those SQL databases: SQL Server, Azure SQL, Postgres, CosmosDB and MySQL. These offer limitless scale, flexible tools, Built-in AI, Performance and Guaranteed availability. 

There are celebrations a foot due to SQL Server on Windows reaching 30 years old this year. SQL Server 2022 is the fastest adopted version of SQL Server with growth on both Windows and Linux. He also mentioned SQL Server enabled by Azure Arc with that cloud manageability for managing your entire SQL Server estate. Announcements were made

Discussions moved to Azure SQL and the family of SQL cloud in edge databases. A further announcement was made about the Azure SQL Managed Instance feature wave , that is now generally available. Azure SQL Managed instance has a free offer in preview for 12 months of free managed instance, available soon.

 What's new in Azure SQL Managed Instance?

They talked about that Cloud Modernization Journey: Evaluate, Optimize, Migrate and Modernize with ease. 

Another announcement Azure SQL Database Hyperscale will be same price as commercial OSS databases – General Available mid December.

Some further to announcements were shared

  • Azure Cosmos DB for MongoDB vCore – Generally Available
  • Vector Search in Azure Cosmos DB for MongoDB vCore – Generally Available 
  • Natural language queries for Azure Cosmos DB – Public Preview
  • Azure Database for PostgreSQL Extension for Azure AI – Preview

Then concluding with the database of the future will be autonomous and intelligent, developer friendly, cost efficient, and mission critical by default.