Data Toboggan Winter Edition 2023 event summary

Data Toboggan winter edition ran on Saturday 28 January 2023. This was our third year and fifth conference. We started with one track of 12 hours and this is the second conference we have had with two tracks of sessions.  So many amazing speakers and sessions of interest. 

Our speakers were from 14 countries this time

USA 10 speakers; Poland 2 speakers; Finland 1 speaker; UK 2 speakers; Germany 2 speakers; Peru 1 speaker; Denmark 2 speakers; India 1 speaker; Sweden 1 speaker; Netherlands 1 speaker; Canada 1 speaker; Brazil 1 speaker ; Uruguay 1 speaker; New Zealand 1 speaker and Australia 1 speaker.

I was pleased to see there were a lot of questions and chat in the rooms and lots of links shared for future reading material.  To help people read up on the links shared, I list them here:

Azure Room

Data Toboggan - YouTube – Watch, subscribe and like.

Data Toboggan Website for news of what is coming next

Data Toboggan | Conference Meetup group for conference registration

Data Toboggan Slide Preparation | User Group

We are putting together categories of the sessions to help people learn Azure Synapse Analytics Learning - Data Toboggan

We were asked several time do we share the videos on the YouTube channel.? We seek approval of the speakers to publish their sessions, and usually aim to publish them around 6 weeks after the event. We can not guarantee all sessions will be available to watch online.

Other learning https://events.sqlbits.com/2023/training-days

The detective agency https://detective.kusto.io/

Recommendation:  T-SQL Fundamantals by Itzik Ben-Gan as an awesome reference

Doing Power BI The Right Way

https://sqlserverbi.blog/presentations/

Analytics Room

Azure Synapse Analytics Migration Guides - Azure Synapse Analytics | Microsoft Learn

Azure Synapse Compatibility Checks - Microsoft Community Hub

Known limitations and issues with Azure Synapse Link for SQL - Azure Synapse Analytics | Microsoft Learn

Dimensional Modeling Techniques - Kimball Group

It is a Data modelling warehouse toolkit, complete star schema reference guide, agile data warehouse design

BEAM Templates — modelstorming.com https://modelstorming.com/templates

Brian Bønk Rueløkke: Slides and code for the Synapse CETAS, Views, Parquet can be found here:

public/Speaks/2023/2023-01-28 Data Toboggan at master · brianbonk/public (github.com)

 Azure Synapse Link for Azure Cosmos DB, benefits, and when to use it | Microsoft Learn

MicrosoftLearning/DP-203-Data-Engineer (github.com)

Azure Synapse Analytics August Update 2022 (microsoft.com)

https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-repair-table.html

Convert databricks noteboooks.dbc into standard .py files - Stack Overflow

Synapse Room

The whitepaper  is good documentation

Azure Synapse Analytics security white paper: Network security

A favourite podcast of ours

Knee-deep in Tech Podcast (kneedeepintech.com)

Certification links:

https://learn.microsoft.com/en-us/certifications/azure-data-fundamentals/

Microsoft Certified: Azure Data Engineer Associate - https://learn.microsoft.com/en-us/certifications/azure-data-engineer/?WT.mc_id=DP-MVP-5004032

Microsoft Certified: Azure Enterprise Data Analyst Associate - https://learn.microsoft.com/en-us/certifications/azure-enterprise-data-analyst-associate/?WT.mc_id=DP-MVP-5004032

Coming soon: The new Azure Enterprise Data Analyst Associate certification

Microsoft ESI -https://esi.microsoft.com/ 

Databricks Academy - https://www.databricks.com/learn/training/login

The exams change regularly, so for any exam it's best to review the study guide. For example here's the DP-203 one;

https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE4MbYT

Blog: https://www.KevinRChant.com
GitHub: https://github.com/kevchant

PastPresentations GitHub repository , which you can find at https://github.com/kevchant/PastPresentations

https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-openrowset for difference in parser versions

Comparison table is here: https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/understand-synapse-dedicated-sql-pool-formerly-sql-dw-and/ba-p/3594628

PVLDB Proceedings Template - WORD -> polaris whitepaper

How to use CETAS on serverless SQL pool to improve performance and automatically recreate it - Microsoft Community Hub

Synapse Spark Delta Time Travel - Microsoft Community Hub

Understand Synapse dedicated SQL pool (formerly SQL DW) and Serverless SQL pool - Microsoft Community Hub

An interesting question to think about: which term will dominate in next 5 years? will it be a data lakehouse? or data mesh? 

Monitoring Synapse serverless SQL pool query history

Monitoring serverless SQL ended requests by using Log Analytics    

I  would like to thank all the speakers for their time and amazing content. So grateful to fellow organisers Andy Cutler and Richard Munn and our moderators,  Justin Bird and Shabnam Watson. And thank you to our attendees for making the day, such a fun day. We are already ready exciting about the next one in the summer.

Data Toboggan 2023 Winter Edition

This is our third year of running Data Toboggan. We are incredibly grateful for all the speakers and attendees who have made this happen. We are pleased to share the agenda. 

Register: https://bit.ly/DT23Register

Agenda:  https://bit.ly/DT23Agenda

Date: Saturday 28th January 2023

Best Practices for using the business glossary in Microsoft Purview

There are many terms that are used within a business and for business to function well it is important to have common terms. There are a few things to think about:

Preparation Standards

• Create a hierarchy of business terms
• Have a naming standard for the glossary. Terms are case sensitive in Purview and allow white space
• Check for Duplicate terms
• Avoid duplicate term names in different parent terms

Term Templates

Using the term template sample.csv file is recommended when importing terms, although a custom template may be used.

Terms updated that already existed will be over written during import, so it is best to test this in a lab environment.

Steward and Experts need to use their Azure Active Directory emails.

Disaster recovery should be considered and exporting the glossary terms could be useful for that.

Multiple Business Glossaries

Multiple business glossaries in Microsoft Purview  were introduced  in December 2022. You can read the article here: Microsoft Purview now supports multiple business glossaries

There are some advantages to this practice.

• To enable different parts of the business to manage the terms with ease
• Where different parts of the business have different needs for a glossary

Glossary terms are not automatically applied to assets. They can be added manually, by bulk edit mode, for up to 25 terms,  or curated code using the Atlas API.

There are various glossary term status:

• Draft – not officially implemented
• Approved – current term
• Expired – no longer used
• Alert – required attention

More details about best practice can be found here

Happy New Year

Happy New Year!

Start the new year with speaking at Data Toboggan 2023. 

CFS is Open : https://sessionize.com/data-toboggan-2023

Event date: 28 January 2023 

A Data Governance Story Part 2

 The second part of the governance story.

Data Catalogues are the foundation stones of data governance  

Data landscape knowledge is a wonderful thing. The data map enables the organisation of data sources by business flow within a security boundary 

Data passed through many different areas being transformed by the rich data he found along the way 

Being able to find and discover new data with ease, is important to bring a data culture to an organisation 

Data provenance can be such an insightful thing 

Automated data classification in tools such as #MicrosoftPurview help speed up insight into data

Managing data sharing between organisations 

A consistent planned route on a map with accurate place location, validated using a compass. The uniqueness of natural geology in a national park complete with a photo & passed the finish line on time 

Data policies for access, security, data residency governing the use of data amongst groups and locations 

Readiness assessments are a great place to start 

Help a business obtain a standard set of commonly used terms across the teams

Data Ethics is a key component of governing data 

Meta data come in many forms  

Establishing a data governance operating model for a village requires all the interconnected parts to come together as a whole for the best outcome 

Data Strategy sets the path to success 

Azure Synapse MVP series video

In the Azure Synapse Analytics month's MVP series video, Ryan Majidimehr was joined by two MVPs, myself and Andy Cutler, to talk about the upcoming Azure Synapse conference called Data Toboggan.  https://msft.it/6019eRKad

A Data Governance Story

 A tale for December. The business and the frameworks.

Time for some cheer 

Datum and Data on a journey. The Data Owner life 

Data working with other stewards

Business is thriving with well governed data 

Data Management Body of Knowledge (DAMA-DMBOK2)  shares the principals of effective data management

The Cloud Data Management Capabilities Framework (CDMC) provides an industry best practice for data management and analytics  

CMMI’s Data Management Maturity (DMM) Model covers best practices for providing support for the implementation of process in 5 categories  

The Data Management Capability Assessment Model (DCAM) can be used for new data management programs or to benchmark  existing programs progress 

The Control of Data Expediently (CODEX) framework allows a birds eye view of the entire system to be taken, to help combine all the elements and environment 

Data having fun juggling and working with different frameworks to cover all elements

Catch the next instalment later.