Welcome

Passionately curious about Data, Databases and Systems Complexity. Data is ubiquitous, the database universe is dichotomous (structured and unstructured), expanding and complex. Find my Database Research at SQLToolkit.co.uk . Microsoft Data Platform MVP

"The important thing is not to stop questioning. Curiosity has its own reason for existing" Einstein



Tuesday, 14 September 2021

Data Governance Podcast

 

It was amazing to take part in my first podcast about the benefits of Data Governance and how to get started in Coeo Conversations with Justin Langford.  

The data field is such an exciting place to be at the moment. Data Governance is more than just compliance, it is about managing the whole ecosystem. I thought I would run out of things to say talking for 30 minutes in the podcast on governance, however that was not the case. I hope you find the podcast interesting, informative and fun.

Looking forward there is an exciting digital event coming up Maximize the Value of Your Data in the Cloud: Achieve unified data governance with Azure Purview . The event is Tuesday, September 28, 2021 | 9:00 AM-10:00 AM Pacific Time (UTC-7) Register here

Monday, 13 September 2021

Data Platform Virtual Summit 2021 Keynote


The Data Platform Summit has started. An excellent session covering the Azure data stack identifying when to use each tool and the key innovations with SQL. It was nice to see an explanation of when to use what tool. With so many tools now it becomes hard to know which is the best choice.  This useful chart was shown.



Azure SQL Edge



SQL Server 2019

Solves the modern data challenges 
  • Data visualization and big data clusters
  • Modern platforms with compatibility
  • Built-in machine learning and extensibility
  • Intelligent performance
  • Layers of security and complain
  • Business critical availability

SQL Linux/Container

Containers are portable and can run anywhere containers are supported. They are lightweight with reduced disk, CPU and memory footprint. They have a consistent image of SQL Server, scripts and tools and are efficient with faster deployment, no patching required and less downtime. 

Azure SQL

SQL Server vs. Azure SQL PasS are 
  • business continuity, high availability, automated backups, long term backup retention , geo-replication
  • scale, advanced security, version-less, built in monitoring and built-in intelligence. 

There are Azure SQL Editions for general purpose, business critical and hyperscale



Azure Arc
Bring Azure data services to on-premises, multi cloud and edge with Azure Arc. Azure Arc enabled SQL Managed Instance has many advantages.




The value of the Cloud provides additional tools such as

  • Azure Defender to protect your data
  • Azure SQL Database Ledger for blockchain
  • Telemetry across all your assets with Azure Monitor SQL Insights

Then to help with migration there are tools available

  • Azure Migrate to discover and assess your SQL Server assets
  • Migrate Inline with Azure Data Studio
  • Migrate online with Azure Database Migration Service or Log Replay Service

A few tools were not mentioned that form part of the data suite such as Azure Cosmos DB. The expansion of tools and options has grown significantly over the last few years so it is always good to assess what business objective you are trying to achieve and select the right tool.
 

Friday, 3 September 2021

Data Strategy: where are we and what is the answer to the ultimate question

Originally published here

Do we know where we are going? Have we asked the right questions? Without a roadmap, we will not arrive at our destination. The first step relies on discovering where we are, what we need to be successful and where we need to go. We need to create a roadmap to enable a path forward. With that roadmap, there is a need to assign owners of tasks throughout the data journey. Data Strategy is a top-down approach closely aligned with business strategy.

Gartner define a 'Data Strategy' as a highly dynamic process employed to support the acquisition, organization, analysis, and delivery of data in support of business objectives. Whereas DAMA defines Data Management as The development, execution, and supervision of plans, policies, programs, and practices that deliver, control, protect, and enhance the value of data and information assets throughout their lifecycles.

Many organisations do not have data strategies in place, although they may be working on areas that would sit under that umbrella. Deciding on what the core data principles are, can help an organisation quickly adapt to the data-centric culture.


As an example a set of data principles could be: 

  • All data is owned, managed, secured, and governed
  • Data is managed throughout its lifecycle
  • Data is available and visible whenever needed
  • Information is an asset
  • Use a data catalogue for visibility
  • Data is fit for purpose and meets the business need
  • There is a single version of the truth
  • Data skills training for people to use data effectively
  • Data ethical standards are followed

 

Before any type of strategy is created a business key stakeholder must champion the idea and a person identified to own the strategy, such as the CDO. The data strategy should be maintained and enacted through the data governance team and other working groups.

Data strategy is a framework that is built around the data to amalgamate the assets to create a source of trusted data to allow process efficiencies, increase confidence in the data and create opportunities for innovation. The Data Management strategy could be aligned with the Data Management Association and the DAMA Body of Knowledge (DMBOK), to enable consistent practices and verifiable decision making.

It is important to have an agile data strategy, thus creating a short-term strategy, so the immediate benefit can be gained by the business. Then working on a longer-term target strategy, once the gap analysis is complete and strategic imperatives are identified.​ A couple of core areas to also review are data governance, data ethics alongside data culture and data skills. The technology side for data collection, data storage, data processing and data output may need updating, but if a need exists for technological change, it will be due to the alignment of business and data strategies and identification throughout the process. To enable that agile approach to data strategy using a Boston matrix with the MoSCoW prioritization technique is very successful.

image003

 

DAMA lists deliverables from strategic planning as:

  • Data Management Charter (Vision, business objectives, guiding principles. success measures, risks, operating mode. A business plan to use the information to create competitive advantage and to support enterprise goals).
  • Data management scope statement (goals and objective for planning, organisation roles, responsibilities clarified)
  • Data Management Implementations Roadmap (programs, projects and tasks, road map and milestones). Requires a data management program strategy a plan for maintaining and improving the quality data integrity access and security and mitigating risks.

Taking all of this into account using systems thinking to gain that holistic view there are three areas that should be covered for success: business data strategy, IT data strategy and operational data strategy.

If you haven’t started creating a data strategy or already have one, it is worth reviewing the current state to ensure an agile actionable plan is in place for continuous improvement.

Friday, 20 August 2021

Maximize the value of your data with Azure Purview

There is a digital event coming on 28 September 2021 9-10am pacific time. The event is about achieving unified data governance with Azure Purview.
Join Microsoft Corporate Vice President Rohan Kumar at this free digital event for demos and deep dives.
Register now to:
  • Learn to create a comprehensive, automated map of all your data.
  • See how Azure Purview works with Azure Synapse Analytics, Power BI, and the rest of your data estate to deliver timely, reliable insights.
  • Watch in-depth demos of product features including Azure Purview Data Map and Data Catalog.
  • Ask Azure experts your data governance questions in the live Q&A.


 

Thursday, 19 August 2021

Azure Purview August updates

There have been some exciting changes to Azure Purview announced. These changes relate to charging and permissions. 

Elastic data maps

The data map is the foundation for data discovery. The data map has two components the throughput created by CRUD operations and storage of the metadata.

The data map can now grow elastically starting at one capacity unit. A capacity unit includes a throughput of 25 operations/sec and the metadata storage scales in increments of 2 GB. Purview Data Map can automatically scale up and down within limits . This new charging model makes Purview much more user friendly and less costly to set up and run.









Access Control in Azure Purview

Note: only applies for Purview accounts created on or after 18 August 18 2021,

A collection is a tool to group assets, sources, and other artifacts into a hierarchy for discoverability and to manage access control. Collections are used to organise and manage assets.

The are various roles that exist:

  • Collection admins - can edit Purview collections, their details, and add sub collections. They can also add users into other Purview roles on collections where they're admins.
  • Data source admins - can manage data sources and data scans.
  • Data curators - can create, read, modify, and delete catalog data assets and set up relationships between assets.
  • Data readers - can access but not modify the data.













There is a great video to watch from W Strasser explaining this new data plane RBAC (role based access control) catalog permission. Being able to fine tune the access to the collections brings with it great advantages. Currently collection names can't be updated or deleted. 

Monday, 9 August 2021

A Summer Retrospective: a bygone era


A few weeks in rural France is just the place to contemplate life and take a step back into a bygone era. An era where there are no phones, no internet and no television. Life can quite easily pass you by and you could go for weeks not speaking to a sole. The fruit on the trees ripen in the orchard, the birds waiting for the perfect moment to swoop and eat the fruit. The roads are mostly empty with the occasional car or logging lorry passing by. Cycling is heaven with the roads to yourself.

This rural area, 214 million years ago, had all life within 300 miles of Rochechouart wiped out when a meteorite, around one of the 15 largest ever to come crashing down on earth. The geological signs of he creator are still present today. This bygone era was also rife with conflict. From the last battle of Richard the 1st - the Lionheart, who laid siege to the Chateau of Chalus-Chabrol, located at the border between Aquitaine and the French kingdom, to the hideouts in the forest of the Maquis du Limousin, who were one of the largest groups of French resistance fighters in the Second World War. The village of Oradour-sur-Glane remains an empty ruin as a memorial for the massacre of its inhabitants.

In this backdrop, technology seems a lifetime away. I can't stress enough the tremendous benefits of taking a technological break for your mental health. You can dream and innovative without the interruption of everyday life.

The age of cloud computing, big data and the algorithm requires a 360-degree perspective. A socio-technical perspective is critical. Reflecting on the changes to the earth, made to this unique landscape from space, you realize that data is in the environment. It is not possible to be an expert in all areas as data is the environment. Data is history, is in the maps, is used in conflict resolution and is used for impact analysis. Data is completely inseparable from life and it drives life, not only business. The  choice of tools available help you navigate through data are vast.

The question is can one truly ever master the entirety of life. Data is life, the past, the present and the future. To truly be a master it requires collaboration, communication and control, as data weaves its interconnected complexity throughout life. A holistic view of this diverse scientific area is required to provide a sustainable future. There is no one best practice that can help navigate this web of graph vertices and edges.

To that end I summise that taking a technological break enables the mind to contemplate and blue sky thinking roam free. Happy Summer break. 

Thursday, 5 August 2021

Data Conferences


I have been surprised by the number of data conferences there are and the continuing growth and diversity of the topics and formats. As well as the main and specialist conferences, a few of which are listed below,  there are a huge number of training events that take place in the evenings and weekends such as  SQL Saturdays ,  Azure Data Community and Data Saturdays . With all these events providing learning opportunities ,and many for free, we are very lucky that we have a community that is so will to share their experience.

Conferences

Date

URL

Dativerse

13-Aug-21

https://datagrillen.com/dativerse/

Data Platform Virtual Summit 

13-18 Sept 2021

https://dataplatformgeeks.com/dps2021/

Future Data Driven 

29-Sep-21

https://datadrivencommunity.com/

DataMinds Connect  

11-12 Oct 2021

https://datamindsconnect.be/

New Stars of Data  

22-Oct-2021

https://www.newstarsofdata.com/

Data Weekender 

06-Nov-2021

https://www.dataweekender.com/

PASS Data Community Summit

8-12 Nov 2021

https://passdatacommunitysummit.com/

The SQL Server & Azure SQL Conference 

7-9 Dec 2021

https://www.mssqlconf.com/#!/

Data & AI Summit 

27-30 June 2022

https://databricks.com/dataaisummit

Big Data LDN

22-23 Sept 2021

https://bigdataldn.com/

SQLBits 

2022

https://sqlbits.com/

Azure Cosmos DB Conf

20-21 April 2021

https://gotcosmos.com/conf

Power BI Summit

7-11 March 2022

https://globalpowerbisummit.com/

Data Toboggan

12-Jun-2021

http://www.datatoboggan.co.uk/

Microsoft Build 2021

25-27 May 2021

https://mybuild.microsoft.com/home

Microsoft Ignite 2021

2-4 Nov 2021

https://myignite.microsoft.com/home

Microsoft Inspire 2021

14-Jul-2021

https://myinspire.microsoft.com/home

DataMinutes

22-Jan-2022

https://datagrillen.com/dataminutes/