PASS 24 Data Community Summit

This week it is PASS 24 Data Community Summit

The Microsoft Keynote was entitled, Fuel AI innovation with Azure Databases.

The session abstract: Data is the fuel for innovation, driving the development of transformative AI applications. In this keynote, join Shireesh and the Microsoft engineering team as they explore harnessing the power of data across the Azure databases portfolio to drive ground breaking advancements. Take your first step on the journey with vector search, multi-agent apps and more and learn to unlock new patterns like retrieval augmented generation (RAG).  Explore the latest database innovations in SQL Server, Azure SQL, Azure Cosmos DB, Azure Database for PostgreSQL and Azure Database for MySQL that enhance operational efficiency, deliver personalized user experiences, and revolutionize the way we interact with technology. 

There were some great announcements 

Announcing

• SSMS 21 and Copilot in SSMS
• Azure SQL Database Hyperscale enhancements (Storage up to 128TB (GA) Log throughput rate up to 150 MB per second (Preview) , Continuous priming (Preview))

Preview

• Next gen general purpose on Azure SQL Managed Instance (32TB of storage, 500 DBs, Lower storage latency, improved storage importance, customisable I/O performance)
• Migration Assessment from SQL Server enabled by Azure Arc
• Mirroring in Fabric for Azure SQL Database and Azure Cosmos DB
• In-database embedding generation
• Vector Indexing and search in Azure Cosmos DB for NoSQL
• Native vector type and functions in Azure SQL Database (Available now)
• DiskANN indexing in Azure Database for PostgreSQL

Generally available

• Bi-directional disaster recovery with link feature in Azure SQL Managed Instance
• Flat and quantized flat vector indexes in Azure Cosmos DB for NoSQL
• Serverless auto-pause delay to 15 min
• Azure AI extension
• Hyperscale elastic pools
• Azure Cosmos DB for MongoDB enhancements

Tech in Ten at SQLBits

Great to see SQLBits post this. 

We keep getting more and more from the team at @venturi_ltd  with their Tech in 10 Podcast - Recorded at SQLBits!

@victoria_holt

from

@CoeoLtd

discussed

@datatoboggan

and more #data-related subjects, as well as SQLBITS! Listen on any podcast platform https://pod.co/tech-in-ten/ep-9-dr-victoria-holt-enterprise-data-architect-coeo

https://pod.co/tech-in-ten/ep-9-dr-victoria-holt-enterprise-data-architect-coeo-9-dr-victoria-holt-enterprise-data-architect-coeo

SQLBits 2022: a hybrid event

What an emotional event. After 2 years of working at home, and everything virtual we all managed to meet in person. It was amazing to meet people in person again at Excel in London from 8 March to 12 March. It came with some apprehension and trying to be careful with testing daily, keeping your social distance as much as possible and some mask wearing as well. I hope this is the first of many more to come, but everyone I spoke to thought things would never return to how they were.  So I think for the time being hybrid events are here to stay.

I was a helper at SQLBits 2022, an orange shirt, helping out with room monitoring and the event. Also with the event being hybrid we also provided virtual room monitoring, chatting and helping those virtual attendees also enjoy the event.  Speakers were both in person and virtual.

Also this year was the start of something new, an initiative to try and help people on their own and first timers enjoy SQLBits.  We are the Bits Buddies.  The central hub was the newly envisioned community corner.  

In addition to that I was part of two panel discussions. 

1. Data Governance with Azure Purview - Ask the Experts with Erwin de Kreuk, Wolfgang Strasser and Richard Munn asking the questions in this open session, there was the chance to ask the panel of 3 MVPs about data governance for your business using Azure Purview.
2. Women in Tech: Keys to a Successful Career Panel Discussion with Rie Merritt, Pragati Jain and Cathrine Wilhelmsen. Our career is a journey that is unique to us, but that doesn't mean we don't have shared experiences. The panel of women data professionals had an open discussion on their winding career paths, how they got here, and where they're heading next. We talked about what it was like to be a woman in tech, what worked, and how building a strong network of people to be in your corner can help you achieve your goals.

I also took part in Venturi's Voice Podcast which focuses on tech, career and leadership. 

It was quiet a juggling act this year but it was a lot of fun and I think something special. I made new friends and the SQLBits organisers were amazing and all those who contributed to the success of the conference.  

SQLBits 2022 Sessions

SQLBits is only a few days away. It seems very strange to be attending an in person event. The SQLBits panel sessions I am taking part in are all about the community pulling together to create sessions. It is amazing that SQLBits have thought a lot about diversity, inclusion and the sqlfamily this year.

Bridge to a new universe: the end-to-end Azure Data Platform

 

Exciting to watch the Day 1 Keynote a Bridge to a new universe: the end-to-end Azure Data Platform delivered by Rohan Kumar and many other people. 

A journey to a new universe is just waiting to inspire innovation, to tap into limitless possibilities and potential. It covered how to shape your data so you can harness its power to find a new galaxy of insights, answers, and predictions.  Some amazing slides and discussion to set you on a new path. 

Three universes bridged together unmatched analytics and insights, limitless cloud data services and unified data governance.

Rohan shared a great quote "If you want to go fast, go alone. If you want to go far, go together. " The SQL Server community has always worked together to achieve some amazing goals. 

Three main data communities were discussed SQLSaturday, Data Saturdays and the Azure Data Community

SQL Server 2022 preview brings with it many new features.  

There are to be two interlinking services Azure Synapse Link and Azure Purview. 

More details were discussed in the keynote but I will share those separately. As ever an inspiring future for us in the data community. 

Read More

Microsoft Ignite book of news

 Announcing SQL Server 2022 preview: Azure-enabled with continued performance and security innovation

Microsoft Ignite November Edition

Microsoft Ignite is back for the second instalment this year. It is November 2–4, 2021 to explore the latest tools, training sessions, technical expertise, networking opportunities, and more. I like to think of Ignite as empowerment and innovation with end to end experiences. The four transformation curated themes 

• Empower everyone for a new world of hybrid work
• Innovate anywhere from multi-cloud to edge
• Build a hyperconnected business
• Protect everything with end-to-end security

Big Data LDN 2021 - Data Governance and the essential CDO

Big Data LDN has been running 22 -23 September. There have been some really interesting sessions covering data governance and the CDO role.

Robin Sutara spoke on the analytics paradox - balancing competing demands for agility and governance. Think big, one technology is not a magic bullet and learn to fail fast. 

Start small with one business problem and build upon that. The data quality with adapt and improve to meet needs of each problem being addressed. It is people and culture that change a business. We need to think beyond the technology. 

 

Great session and an important thought for the future to stop talking architecture and focus on business value. It goes beyond economics and needs sustainability.

Data is a hot mess, so lets cook session from Ben Schein mentioned a key point Data Governance is never done and always be open to new ideas. Data governance is a team sport. You need to find a balance between innovation and experimentation, data teams and consistency, usability and scale. He finished with some important areas to consider

• keep data definitions consistent
• don't wait for  perfect data set to arrive
• data quality is essential for good insights
• let people go shopping for the data they needs
• there is no data magic wand
• data governance is a team sport

Another very insightful session from Cindi Howson on data-driven culture: is your organisation a laggard or leader, gave perspectives on the biggest challenges to being data driven. 

Perspectives on the biggest challenge to being data driven is culture (67%) followed by talent and people (22%).

 

To disrupt your culture bring in a change agent, identify relevance (WIIFM) and organise for collaboration.

She talked about CDOs having a short tenure in organizations being an average of 2.5 years. There is an interesting article about that , Why Do Chief Data Officers Have Such Short Tenures? by Tom Davenport, Randy Bean, and Josh King 

Why WIIFM (what's in it for me) for communication, incentives, skills and tribes and role models.

To move forward into this new world there needs to be a transition in organizational design, from that traditional BI centre of  excellence to this embedded design .

Centralize for economies of scale for common data, infrastructure and specialist talent but decentralize when business domain experts are essential to analytics workflow. Ensure that cross boundary communication is ongoing for best practice, synergies and career management. We are looking at a new hybrid model that is transparent and optimized.

She finished with sharing there is a Data Chief Community (TheDataChief.com) . It has podcasts, blog,  roundtables and community newsletter. 

What we learned from 400 data leaders in CDO summer school  session by Carruthers and Jackson highlighted

• Data strategy is multi dimensional not linear
• Governance is being revised but you need to tell the story. It is still a problem but it needs communicating the purpose through strategy.
• They had Scott Taylor talk about data storytelling - keep simple
• Risk - listen, listen and listen more 
• Soft skills are important
• Data Literacy is a spectrum.
• Nurture your community
• Play to your strengths and culture is the biggest hurdle
• Keep policy simple
• There should be a call to arms on Ethics
• Methods can be disrupted, innovative and evoke change
• It is not about technology outcomes

In the summer school they discussed the DIKW Pyramid which represents the relationships between data, information, knowledge and wisdom

Exasol have published the journey to the CDO. 

https://www.thedatadreamer.com/insights/the-journey-to-CDO/

Data Platform Virtual Summit 2021 Keynote

The Data Platform Summit has started. An excellent session covering the Azure data stack identifying when to use each tool and the key innovations with SQL. It was nice to see an explanation of when to use what tool. With so many tools now it becomes hard to know which is the best choice.  This useful chart was shown.

Azure SQL Edge

SQL Server 2019

Solves the modern data challenges 

• Data visualization and big data clusters
• Modern platforms with compatibility
• Built-in machine learning and extensibility
• Intelligent performance
• Layers of security and complain
• Business critical availability

SQL Linux/Container

Containers are portable and can run anywhere containers are supported. They are lightweight with reduced disk, CPU and memory footprint. They have a consistent image of SQL Server, scripts and tools and are efficient with faster deployment, no patching required and less downtime. 

Azure SQL

SQL Server vs. Azure SQL PasS are 

• business continuity, high availability, automated backups, long term backup retention , geo-replication
• scale, advanced security, version-less, built in monitoring and built-in intelligence. 

There are Azure SQL Editions for general purpose, business critical and hyperscale

Azure Arc

Bring Azure data services to on-premises, multi cloud and edge with Azure Arc. Azure Arc enabled SQL Managed Instance has many advantages.

The value of the Cloud provides additional tools such as

• Azure Defender to protect your data
• Azure SQL Database Ledger for blockchain
• Telemetry across all your assets with Azure Monitor SQL Insights

Then to help with migration there are tools available

• Azure Migrate to discover and assess your SQL Server assets
• Migrate Inline with Azure Data Studio
• Migrate online with Azure Database Migration Service or Log Replay Service

A few tools were not mentioned that form part of the data suite such as Azure Cosmos DB. The expansion of tools and options has grown significantly over the last few years so it is always good to assess what business objective you are trying to achieve and select the right tool.
 

Maximize the value of your data with Azure Purview

There is a digital event coming on 28 September 2021 9-10am pacific time. The event is about achieving unified data governance with Azure Purview.
Join Microsoft Corporate Vice President Rohan Kumar at this free digital event for demos and deep dives.
Register now to:

• Learn to create a comprehensive, automated map of all your data.

• See how Azure Purview works with Azure Synapse Analytics, Power BI, and the rest of your data estate to deliver timely, reliable insights.

• Watch in-depth demos of product features including Azure Purview Data Map and Data Catalog.

• Ask Azure experts your data governance questions in the live Q&A.

 

Data Conferences

I have been surprised by the number of data conferences there are and the continuing growth and diversity of the topics and formats. As well as the main and specialist conferences, a few of which are listed below,  there are a huge number of training events that take place in the evenings and weekends such as  SQL Saturdays ,  Azure Data Community and Data Saturdays . With all these events providing learning opportunities ,and many for free, we are very lucky that we have a community that is so will to share their experience.

Conferences	Date	URL
Dativerse	13-Aug-21	https://datagrillen.com/dativerse/
Data Platform Virtual Summit	13-18 Sept 2021	https://dataplatformgeeks.com/dps2021/
Future Data Driven	29-Sep-21	https://datadrivencommunity.com/
DataMinds Connect	11-12 Oct 2021	https://datamindsconnect.be/
New Stars of Data	22-Oct-2021	https://www.newstarsofdata.com/
Data Weekender	06-Nov-2021	https://www.dataweekender.com/
PASS Data Community Summit	8-12 Nov 2021	https://passdatacommunitysummit.com/
The SQL Server & Azure SQL Conference	7-9 Dec 2021	https://www.mssqlconf.com/#!/
Data & AI Summit	27-30 June 2022	https://databricks.com/dataaisummit
Big Data LDN	22-23 Sept 2021	https://bigdataldn.com/
SQLBits	2022	https://sqlbits.com/
Azure Cosmos DB Conf	20-21 April 2021	https://gotcosmos.com/conf
Power BI Summit	7-11 March 2022	https://globalpowerbisummit.com/
Data Toboggan	12-Jun-2021	http://www.datatoboggan.co.uk/
Microsoft Build 2021	25-27 May 2021	https://mybuild.microsoft.com/home
Microsoft Ignite 2021	2-4 Nov 2021	https://myignite.microsoft.com/home
Microsoft Inspire 2021	14-Jul-2021	https://myinspire.microsoft.com/home
DataMinutes	22-Jan-2022	https://datagrillen.com/dataminutes/

Data Toboggan Cool Runnings 21 Summary

The 12 hour event ran on 12 June. We used Teams Live Events. The tool has evolved since we used it in January with some plus points and some behaviour we hadn't seen before. There were some amazing speakers from around the world: Finland, Malta, Australia, Hungary, Serbia, UK (including Scotland), The Netherlands, USA (Seattle, New Mexico), China, (Shanghai)  Canada and Norway. 

It was amazing having Robin Sutara, the Microsoft UK Chief Data Officer present the Keynote. We were also privileged to have a number of product group speakers sharing in depth details about the Azure Synapse features. Then there were community speakers and MVPs speaking. All in all a lot of content with the 45 minutes main sessions wrapped with 5 minute pre recorded lightning talks.  

We had our first expert panel discussion session which I found really interesting and informative. People were able to ask questions in advance using Microsoft forms and during the event using Slido. I would like to thank the people who watched the presentations live on the day and contributed to such a lively chat on Slack. 

I was interested to see where our twitter followers are located. It is interesting to see where Azure Synapse is being used or investigated.

There were many discussions to be had by people on the slack channel. It is nice to have free open discussions on Azure Synapse all day. The links shared in the slack channel were

Presentation tricks : http://blogs.lobsterpot.com.au/2021/01/30/presentation-trickery-online-glassboard-like-lightboard-but-using-just-free-software/ 

Set the utf 8 collation after the dB has been created described by Jovan here https://techcommunity.microsoft.com/t5/azure-synapse-analytics/always-use-utf-8-collations-to-read-utf-8-text-in-serverless-sql/ba-p/1883633

Learn more about distributed execution flow that serverless uses, you can read it in this VLDB article: https://www.vldb.org/pvldb/vol13/p3204-saborit.pdf

Andy C Slides for Turbocharge here https://www.datahai.co.uk/power-bi/turbocharge-power-bi-using-azure-synapse-analytics-session/

Using file metadata in queries : https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/query-specific-files

Best Practices for serverless SQL pool https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/best-practices-serverless-sql-pool

Article that explains the cost management for serverless pools in details: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/data-processed

Synapse Link for Dataverse: https://docs.microsoft.com/en-us/powerapps/maker/data-platform/export-to-data-lake

Roman Pijack: Cost control for serverless SQL pool - the source code of the views I'll be showing during the lightning talk is available   https://github.com/RomanPijacek/DataToboggan/tree/main/CostControlForServerlessSqlPool

The official MS doc that describes Cost management for serverless SQL pool: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/data-processed

Craig Porteous: github repo  for his session 

https://github.com/cporteou/Presentations/tree/main/Adventures%20in%20CICD%20with%20Azure%20Synapse

Data Profiler Summary Stats in ADF Data Flows: https://techcommunity.microsoft.com/t5/azure-data-factory/how-to-save-your-data-profiler-summary-stats-in-adf-data-flows/ba-p/1243251

Data Factory to Synapse user voice: https://feedback.azure.com/forums/307516-azure-synapse-analytics/suggestions/41026642-allow-to-share-a-self-hosted-ir-from-data-factory

Well-Architected Framework session slides  https://www.datahai.co.uk/synapse-analytics/applying-the-azure-well-architected-framework-to-azure-synapse-analytics-session/

Mark PM: SQLPackage.exe https://docs.microsoft.com/en-us/sql/tools/sqlpackage/sqlpackage?view=sql-server-ver15 

SqlPackage for Azure Synapse Analytics - SQL Server https://docs.microsoft.com/en-us/sql/tools/sqlpackage/sqlpackage-for-azure-synapse-analytics?view=sql-server-ver15

Wolfgang: First preview this Summer https://devblogs.microsoft.com/visualstudio/visual-studio-2022/

From Drew Skwiers-Koballa  "Build a Data Warehouse with SQL Database Projects": https://github.com/dzsquared/synapse-sqlproj-demo

Finishing with the Azure Synapse Analytics Blog: https://techcommunity.microsoft.com/t5/azure-synapse-analytics/bg-p/AzureSynapseAnalyticsBlog 

Data Toboggan - Cool Runnings line up

We have an amazing line up of speakers and subject matter experts. 

We have a a lot of different elements with the summer edition of data toboggan.

• Our keynote will be delivered by Robin Sutara the Microsoft UK Chief Data Officer. I'm really excited to have the opportunity to listen to Robin kick off out event  
• Ask the Experts panel from Microsoft and the Community
• A Cloud Skills Challenge on Azure Synapse between 1st June 2021 and ends 23rd June 2021. The Cloud Skills Challenge is here for UK residents and the terms and conditions are available here 
• Members of the Microsoft Product group sharing their knowledge of the latest features and more
• MVPs and community members who are passionate about Azure Synapse
• Lightning talks for a light break between sessions. We have some fun sessions, short topics and interesting research talks.  
• And most importantly you the community attending and sharing in our excitement for Azure Synapse.

We hope you enjoy the day around the globe on Saturday 12 June 8am (BST) to 8pm (BST).  

Register now: http://bit.ly/DT-CRRegister 

Agenda: https://bit.ly/DT-CRAgenda

#AzureSynapse @Azure_Synapse #MVPBuzz

Read: Azure Synapse: A Single Pane of Glass

Data Toboggan - Ask the Experts

Data Toboggan are really excited to have an Ask the Experts from Microsoft and the Community session. Submit your questions now http://bit.ly/DTAsktheExperts Register http://bit.ly/DT-CRRegister

The experts

Wee Hyong Tok (@weehyong), Principal Group Program Manager at Microsoft Shirley Wang, Principal Group Program Manager at Microsoft Mark Kromer (@KromerBigData), Principal Program Manager at Microsoft Abhishek Narrain (@narainabhishek), Senior Program Manager at Microsoft Linda Wang, Program Manager at Microsoft Leslie Andrews (@landrews5807), Lead Data Architect at 3Cloud Rayis Imayev (@RayisImayev), Senior Data Engineer, Global Investments at OMERS Kamil Nowinski (@NowinskiK), Group Manager & Analytics Architect at Avanade UK

See the post by Cathrine Wilhelmsen. https://bit.ly/3fO9aO7

To take part in an amazing day of sessions for 12 hours of Azure Synapse on Saturday 12 June register now: http://bit.ly/DT-CRRegister

Agenda: https://bit.ly/DT-CRAgenda