Welcome

Passionately curious about Data, Databases and Systems Complexity. Data is ubiquitous, the database universe is dichotomous (structured and unstructured), expanding and complex. Find my Database Research at SQLToolkit.co.uk . Microsoft Data Platform MVP

"The important thing is not to stop questioning. Curiosity has its own reason for existing" Einstein



Monday, 13 September 2021

Data Platform Virtual Summit 2021 Keynote


The Data Platform Summit has started. An excellent session covering the Azure data stack identifying when to use each tool and the key innovations with SQL. It was nice to see an explanation of when to use what tool. With so many tools now it becomes hard to know which is the best choice.  This useful chart was shown.



Azure SQL Edge



SQL Server 2019

Solves the modern data challenges 
  • Data visualization and big data clusters
  • Modern platforms with compatibility
  • Built-in machine learning and extensibility
  • Intelligent performance
  • Layers of security and complain
  • Business critical availability

SQL Linux/Container

Containers are portable and can run anywhere containers are supported. They are lightweight with reduced disk, CPU and memory footprint. They have a consistent image of SQL Server, scripts and tools and are efficient with faster deployment, no patching required and less downtime. 

Azure SQL

SQL Server vs. Azure SQL PasS are 
  • business continuity, high availability, automated backups, long term backup retention , geo-replication
  • scale, advanced security, version-less, built in monitoring and built-in intelligence. 

There are Azure SQL Editions for general purpose, business critical and hyperscale



Azure Arc
Bring Azure data services to on-premises, multi cloud and edge with Azure Arc. Azure Arc enabled SQL Managed Instance has many advantages.




The value of the Cloud provides additional tools such as

  • Azure Defender to protect your data
  • Azure SQL Database Ledger for blockchain
  • Telemetry across all your assets with Azure Monitor SQL Insights

Then to help with migration there are tools available

  • Azure Migrate to discover and assess your SQL Server assets
  • Migrate Inline with Azure Data Studio
  • Migrate online with Azure Database Migration Service or Log Replay Service

A few tools were not mentioned that form part of the data suite such as Azure Cosmos DB. The expansion of tools and options has grown significantly over the last few years so it is always good to assess what business objective you are trying to achieve and select the right tool.
 

Friday, 3 September 2021

Data Strategy: where are we and what is the answer to the ultimate question

Originally published here

Do we know where we are going? Have we asked the right questions? Without a roadmap, we will not arrive at our destination. The first step relies on discovering where we are, what we need to be successful and where we need to go. We need to create a roadmap to enable a path forward. With that roadmap, there is a need to assign owners of tasks throughout the data journey. Data Strategy is a top-down approach closely aligned with business strategy.

Gartner define a 'Data Strategy' as a highly dynamic process employed to support the acquisition, organization, analysis, and delivery of data in support of business objectives. Whereas DAMA defines Data Management as The development, execution, and supervision of plans, policies, programs, and practices that deliver, control, protect, and enhance the value of data and information assets throughout their lifecycles.

Many organisations do not have data strategies in place, although they may be working on areas that would sit under that umbrella. Deciding on what the core data principles are, can help an organisation quickly adapt to the data-centric culture.


As an example a set of data principles could be: 

  • All data is owned, managed, secured, and governed
  • Data is managed throughout its lifecycle
  • Data is available and visible whenever needed
  • Information is an asset
  • Use a data catalogue for visibility
  • Data is fit for purpose and meets the business need
  • There is a single version of the truth
  • Data skills training for people to use data effectively
  • Data ethical standards are followed

 

Before any type of strategy is created a business key stakeholder must champion the idea and a person identified to own the strategy, such as the CDO. The data strategy should be maintained and enacted through the data governance team and other working groups.

Data strategy is a framework that is built around the data to amalgamate the assets to create a source of trusted data to allow process efficiencies, increase confidence in the data and create opportunities for innovation. The Data Management strategy could be aligned with the Data Management Association and the DAMA Body of Knowledge (DMBOK), to enable consistent practices and verifiable decision making.

It is important to have an agile data strategy, thus creating a short-term strategy, so the immediate benefit can be gained by the business. Then working on a longer-term target strategy, once the gap analysis is complete and strategic imperatives are identified.​ A couple of core areas to also review are data governance, data ethics alongside data culture and data skills. The technology side for data collection, data storage, data processing and data output may need updating, but if a need exists for technological change, it will be due to the alignment of business and data strategies and identification throughout the process. To enable that agile approach to data strategy using a Boston matrix with the MoSCoW prioritization technique is very successful.

image003

 

DAMA lists deliverables from strategic planning as:

  • Data Management Charter (Vision, business objectives, guiding principles. success measures, risks, operating mode. A business plan to use the information to create competitive advantage and to support enterprise goals).
  • Data management scope statement (goals and objective for planning, organisation roles, responsibilities clarified)
  • Data Management Implementations Roadmap (programs, projects and tasks, road map and milestones). Requires a data management program strategy a plan for maintaining and improving the quality data integrity access and security and mitigating risks.

Taking all of this into account using systems thinking to gain that holistic view there are three areas that should be covered for success: business data strategy, IT data strategy and operational data strategy.

If you haven’t started creating a data strategy or already have one, it is worth reviewing the current state to ensure an agile actionable plan is in place for continuous improvement.

Friday, 20 August 2021

Maximize the value of your data with Azure Purview

There is a digital event coming on 28 September 2021 9-10am pacific time. The event is about achieving unified data governance with Azure Purview.
Join Microsoft Corporate Vice President Rohan Kumar at this free digital event for demos and deep dives.
Register now to:
  • Learn to create a comprehensive, automated map of all your data.
  • See how Azure Purview works with Azure Synapse Analytics, Power BI, and the rest of your data estate to deliver timely, reliable insights.
  • Watch in-depth demos of product features including Azure Purview Data Map and Data Catalog.
  • Ask Azure experts your data governance questions in the live Q&A.


 

Thursday, 19 August 2021

Azure Purview August updates

There have been some exciting changes to Azure Purview announced. These changes relate to charging and permissions. 

Elastic data maps

The data map is the foundation for data discovery. The data map has two components the throughput created by CRUD operations and storage of the metadata.

The data map can now grow elastically starting at one capacity unit. A capacity unit includes a throughput of 25 operations/sec and the metadata storage scales in increments of 2 GB. Purview Data Map can automatically scale up and down within limits . This new charging model makes Purview much more user friendly and less costly to set up and run.









Access Control in Azure Purview

Note: only applies for Purview accounts created on or after 18 August 18 2021,

A collection is a tool to group assets, sources, and other artifacts into a hierarchy for discoverability and to manage access control. Collections are used to organise and manage assets.

The are various roles that exist:

  • Collection admins - can edit Purview collections, their details, and add sub collections. They can also add users into other Purview roles on collections where they're admins.
  • Data source admins - can manage data sources and data scans.
  • Data curators - can create, read, modify, and delete catalog data assets and set up relationships between assets.
  • Data readers - can access but not modify the data.













There is a great video to watch from W Strasser explaining this new data plane RBAC (role based access control) catalog permission. Being able to fine tune the access to the collections brings with it great advantages. Currently collection names can't be updated or deleted. 

Monday, 9 August 2021

A Summer Retrospective: a bygone era


A few weeks in rural France is just the place to contemplate life and take a step back into a bygone era. An era where there are no phones, no internet and no television. Life can quite easily pass you by and you could go for weeks not speaking to a sole. The fruit on the trees ripen in the orchard, the birds waiting for the perfect moment to swoop and eat the fruit. The roads are mostly empty with the occasional car or logging lorry passing by. Cycling is heaven with the roads to yourself.

This rural area, 214 million years ago, had all life within 300 miles of Rochechouart wiped out when a meteorite, around one of the 15 largest ever to come crashing down on earth. The geological signs of he creator are still present today. This bygone era was also rife with conflict. From the last battle of Richard the 1st - the Lionheart, who laid siege to the Chateau of Chalus-Chabrol, located at the border between Aquitaine and the French kingdom, to the hideouts in the forest of the Maquis du Limousin, who were one of the largest groups of French resistance fighters in the Second World War. The village of Oradour-sur-Glane remains an empty ruin as a memorial for the massacre of its inhabitants.

In this backdrop, technology seems a lifetime away. I can't stress enough the tremendous benefits of taking a technological break for your mental health. You can dream and innovative without the interruption of everyday life.

The age of cloud computing, big data and the algorithm requires a 360-degree perspective. A socio-technical perspective is critical. Reflecting on the changes to the earth, made to this unique landscape from space, you realize that data is in the environment. It is not possible to be an expert in all areas as data is the environment. Data is history, is in the maps, is used in conflict resolution and is used for impact analysis. Data is completely inseparable from life and it drives life, not only business. The  choice of tools available help you navigate through data are vast.

The question is can one truly ever master the entirety of life. Data is life, the past, the present and the future. To truly be a master it requires collaboration, communication and control, as data weaves its interconnected complexity throughout life. A holistic view of this diverse scientific area is required to provide a sustainable future. There is no one best practice that can help navigate this web of graph vertices and edges.

To that end I summise that taking a technological break enables the mind to contemplate and blue sky thinking roam free. Happy Summer break. 

Thursday, 5 August 2021

Data Conferences


I have been surprised by the number of data conferences there are and the continuing growth and diversity of the topics and formats. As well as the main and specialist conferences, a few of which are listed below,  there are a huge number of training events that take place in the evenings and weekends such as  SQL Saturdays ,  Azure Data Community and Data Saturdays . With all these events providing learning opportunities ,and many for free, we are very lucky that we have a community that is so will to share their experience.

Conferences

Date

URL

Dativerse

13-Aug-21

https://datagrillen.com/dativerse/

Data Platform Virtual Summit 

13-18 Sept 2021

https://dataplatformgeeks.com/dps2021/

Future Data Driven 

29-Sep-21

https://datadrivencommunity.com/

DataMinds Connect  

11-12 Oct 2021

https://datamindsconnect.be/

New Stars of Data  

22-Oct-2021

https://www.newstarsofdata.com/

Data Weekender 

06-Nov-2021

https://www.dataweekender.com/

PASS Data Community Summit

8-12 Nov 2021

https://passdatacommunitysummit.com/

The SQL Server & Azure SQL Conference 

7-9 Dec 2021

https://www.mssqlconf.com/#!/

Data & AI Summit 

27-30 June 2022

https://databricks.com/dataaisummit

Big Data LDN

22-23 Sept 2021

https://bigdataldn.com/

SQLBits 

2022

https://sqlbits.com/

Azure Cosmos DB Conf

20-21 April 2021

https://gotcosmos.com/conf

Power BI Summit

7-11 March 2022

https://globalpowerbisummit.com/

Data Toboggan

12-Jun-2021

http://www.datatoboggan.co.uk/

Microsoft Build 2021

25-27 May 2021

https://mybuild.microsoft.com/home

Microsoft Ignite 2021

2-4 Nov 2021

https://myignite.microsoft.com/home

Microsoft Inspire 2021

14-Jul-2021

https://myinspire.microsoft.com/home

DataMinutes

22-Jan-2022

https://datagrillen.com/dataminutes/


Monday, 2 August 2021

Responsible Innovation: A Best Practices Toolkit

Responsible innovation is a toolkit that helps developers become good stewards for the future of science and its effect on society.  

There are 3 areas
  • Judgment Call
  • Harms Modelling
  • Community Jury

This toolkit provides a set of practices currently in development, for anticipating and addressing the potential negative impacts of technology on people. This is an early release of this development.

Judgment Call 

Judgment Call is an award-winning game and team-based activity that puts Microsoft’s AI principles of fairness, privacy and security, reliability and safety, transparency, inclusion, and accountability into action. The game cultivates stakeholder empathy through scenario-imagining. Game participants write product reviews from the perspective of a particular stakeholder, describing what kind of impact and harms the technology could produce from their point of view.

To prepare for this game, download the printable Judgment Call game kit.























Harms Modelling 

Harms Modelling is a framework for product teams, grounded in four core pillars of responsible innovation, that examine how people's lives can be negatively impacted by technology: injuries, denial of consequential services, infringement on human rights, and erosion of democratic & societal structures. Similar to Security Threat Modelling, This modelling enables product teams to anticipate potential real-world impacts of technology.



Community Jury

Community Jury is a technique that brings together diverse stakeholders impacted by a technology. It is an adaptation of the citizen jury. The stakeholders are provided an opportunity to learn from experts about a project, deliberate together, and give feedback on use cases and product design. This responsible innovation technique allows project teams to collaborate with researchers to identify stakeholder values, and understand the perceptions and concerns of impacted stakeholders.

These 3  new tools under development are underdevelopment but quiet interesting to look at. 

References

Citizens Juries

The Ethics of AI Ethics: An Evaluation of Guidelines

Hagendorff, T. The Ethics of AI Ethics: An Evaluation of Guidelines. Minds & Machines 30, 99–120 (2020)