The Data Saturday Oslo Event

I was really excited to go and speak at Data Saturday in Oslo on #35 on 2 September 2023 on the Business Benefits for good governance.  Oslo is an amazing city and the Saturday venue was in the heart of the Campus Fjerdingen | Høyskolen Kristiania, Chr. Krohgs gate 32, 0186 Oslo

It was great to see all the room with Norse names

I spoke after lunch at 14:15 in Vanaheim, one of the nine worlds that are situated around the world tree Yggdrasil.

My session slides are on Github

The event was packed full of amazing session content with the day being a true community event.  I would recommend the event to anyone in the future and hope i have to opportunity to speak there again in the future. 

Why Data Quality is important for your business

Original article I wrote was Published on LinkedIn 

Data Quality

#data volumes are increasing significantly. It is hard to keep track of the data used in a business and to know how accurate the data is, that is being used for business decisions. Accurate data is critical to the success of a business. Ensuring that data comes from the right source, is not duplicated, does not have syntax issues or is incorrectly updated requires good data management.

What is data quality?

DAMA UK defines data quality as ‘the planning , implementation , and control of activities that apply quality management techniques to data, in order to assure it is fit for consumption and meets the needs of data consumers’ So good data quality needs accurate data that meets the needs of a business.

There is also an international standard for data quality, ISO 8000. This states ‘Quality is actually the conformance of characteristics to requirements and, thus, any item of data can be of high quality for one use but not for another use that has differing requirements.’ The ISO8000 identifies that data quality has syntactic (format), semantic (meaning) and pragmatic (usefulness) characteristics.

Why is data quality important

Good data quality enables strong business decisions to be made leading to better business outcomes. These business decisions based on data can lead to greater profitability, helping improve situations and can be the starting place for predictive analytics using AI systems. Poor data quality often occurs due to human error and being able to put checks and balances in place to know the current state of the data is important.

Measuring data quality

It is important for every business to understand the level of their data quality maturity. This enables trustworthy decisions to be made and can even impact data ethical understanding. There are 6 main dimensions to consider.

• Completeness – this metric addresses the requirements that all data sets and data items have all the relevant information. The measure is about that missing information, thinking about the proportion of data received, incomplete, missing, and data loss.
• Accuracy – does the data reflect the data set real world, is it truthful containing the correct data entries.
• Uniqueness – is a single view of the data. This metric looks at the extent of the duplication of data, with consideration of how data is controlled.
• Validity – does the data match the rules such as syntax (format, type, range) of its definition.
• Consistency – do the data sets match across data stored in two or more records. Does the pattern \ frequency of data match.
• Timeliness – the degree the data represents reality at a point in time. Is the data available when required.

DAMA provide details on how to calculate these measure to provide those KPI’s to business. A useful technique to help provide a view on the current state of the data, is data profiling. This can look at things like counting nulls in data, the max/min value, max/min length, frequency distribution and data type and format.

Data enhancements

The main take away is to always consider the quality of the data you use and embed checks into the business processes. Always have a dashboard showing the current state of business data quality. If you are looking for a framework to help guide your thought processes the UK government has created a data management framework which is worth reviewing. In conclusion start with a review of the data quality of the core data sources you use in your data catalog and record the current state.

Key next steps to improve Data Quality

To start on the path of immediately improving the data quality, for the organisation and enable better analytics:

• Understand what data exists

• Assign a relevant data owner to the data sources

• For each business use case understand the lineage of the data used for reports and dashboards.

Here are some helpful resources to assist with the above steps!

Creating a Data Catalog with Microsoft Purview - Cloud Adoption Framework

Improving Data Lineage with Microsoft Purview - Cloud Adoption Framework

Data quality considerations - Cloud Adoption Framework

A Summer Retrospective: a bygone era

A few weeks in rural France is just the place to contemplate life and take a step back into a bygone era. An era where there are no phones, no internet and no television. Life can quite easily pass you by and you could go for weeks not speaking to a sole. The fruit on the trees ripen in the orchard, the birds waiting for the perfect moment to swoop and eat the fruit. The roads are mostly empty with the occasional car or logging lorry passing by. Cycling is heaven with the roads to yourself.

This rural area, 214 million years ago, had all life within 300 miles of Rochechouart wiped out when a meteorite, around one of the 15 largest ever to come crashing down on earth. The geological signs of he creator are still present today. This bygone era was also rife with conflict. From the last battle of Richard the 1st - the Lionheart, who laid siege to the Chateau of Chalus-Chabrol, located at the border between Aquitaine and the French kingdom, to the hideouts in the forest of the Maquis du Limousin, who were one of the largest groups of French resistance fighters in the Second World War. The village of Oradour-sur-Glane remains an empty ruin as a memorial for the massacre of its inhabitants.

In this backdrop, technology seems a lifetime away. I can't stress enough the tremendous benefits of taking a technological break for your mental health. You can dream and innovative without the interruption of everyday life.

The age of cloud computing, big data and the algorithm requires a 360-degree perspective. A socio-technical perspective is critical. Reflecting on the changes to the earth, made to this unique landscape from space, you realize that data is in the environment. It is not possible to be an expert in all areas as data is the environment. Data is history, is in the maps, is used in conflict resolution and is used for impact analysis. Data is completely inseparable from life and it drives life, not only business. The  choice of tools available help you navigate through data are vast.

The question is can one truly ever master the entirety of life. Data is life, the past, the present and the future. To truly be a master it requires collaboration, communication and control, as data weaves its interconnected complexity throughout life. A holistic view of this diverse scientific area is required to provide a sustainable future. There is no one best practice that can help navigate this web of graph vertices and edges.

To that end I summise that taking a technological break enables the mind to contemplate and blue sky thinking roam free. Happy Summer break. 

Microsoft MVP for a third year

During such a life changing time, I am over the moon with joy to be honoured with my third Microsoft Most Valuable Professional (MVP) award in recognition of exceptional technical community leadership for Data Platform for 2020-2021. How amazing to receive this. Thank you @MVPAward #MVPBuzz . There are so many amazing MVP's who help the community improve and grow their data platform knowledge. I am passionate about helping share the knowledge and experience I have gained.

This has been a strange period with many events cancelled and turned into virtual events. As an organizer of  Data Relay this year we cancelled our event to make way for SQLBits to move into the only viable slot this year. As things are still ongoing I think it is unlikely any in person conferences will continue this year.  However there are some amazing free online events to attend. My thoughts have been on helping and supporting the local community as well as the data community. 

Christmas BINGO

Here is something to keep you amused whilst reading those technical articles over Christmas.

How many articles do you have to read to say BINGO?

T: @victoria_holt

Shared and certified datasets

Microsoft certified data sets shared at the Buisness Applications Summit, will discover and reuse trusted data assets in your organization.

Women who are shining lights in our city

Proud to be in this article in the Bath Chronicle promoting STEM in the West of England.


htag

Understanding Complexity with Systems

I have today been watching the many butterflies flapping their wings around a Buddleia bush. The butterfly bush with its abundant deep violet-purple flowers attracts the butterflies, which surround the bush. I always wonder if this activity will cause a transformation round the globe with the much discussed 'butterfly effect'. The classic quote 'the notion that a butterfly stirring the air today in Peking can transform storm systems next month in New York [...] tiny differences in input could quickly become overwhelming differing in output' Glick (1987) . This is known as Chaos,  the term coined by Lorenz for a system with unpredictable outcomes.  The problem is that it is difficult to know the exact starting point of a situation accurately enough to put it into a mathematical formula. Thus each step in the process in the system moves further away from where you thought it should go. The errors or uncertainty multiply and can cause turbulent features. 

The natural world is unpredictable and has many patterns of behaviour. To understand any system well enough it is necessary to understand the inputs and outputs. To understand database systems there is much complexity that needs to be considered. This graph shows some of the theoretical components.

To apply this understanding and advance a system using artificial  intelligence (AI) you need to fully understand the system under investigation. For data and database systems people gain that experience and learning over many years. The art is to document the inputs and outputs and identify the best practices which have worked and those that have not. Also to create a method to connect the continuously evolving and changing best practice. Only then can you identify the opportunities within the system to improve management and create a state where AI is embedded within a helpful tool that could improve efficiency and performance.

My research examined the complexity of managing database systems and as such has the building blocks to begin to build an autonomous AI system to help manage database systems. Below shows the components that are interconnected in the the management of database systems.

Complexity is always changing and migrating with the passage of time and the trend will be from order to disorder, thus creating an autonomous system to help prevent that could be beneficial.The aim would be to create a self organizing system based on feedback from a persons behaviour, decisions, documentation, operational configuration, meetings and actions. 

2018 MVP Reactions

Peter Laker published an article about reactions from the 2018's NEWEST Most Valuable Professional (MVP) award winners. What a great set of reactions from some amazing people.

Privileged to have my comment listed on the reactions list.

Azure CosmosDB Change Feed

The Azure CosmosDB change feed can provide a persistent log of records within an Azure CosmosDB container. You can learn about this from this concise presentation.

Big Data Exploration

The big data landscape is growing and exploration of the data can help make better decisions. I came across this great infographic from IBM.

Literature Map

When you start any research project, you need to set the research in the context of the current literature. This will establish a framework for the importance of the study. This document was the starting place for organizing the literature of interest in my research.

Thesis Title: A Study in Best Practices and Procedures for the Management of Database Systems

Data past, data present and data future

Database and data management practices are changing and new practices need to be adaptive and agile. Here are a few data thoughts for data past, data present and data future.

Data of Christmas Past

•  Reports and dashboards have become standard place within the business. 
•  Big data is just the new look of data.
• DevOps has become mainstream.

Data of Christmas Present

• There is an increasing plethora of database architecture designs to choose from which means selecting the right design and database engine for the right job is harder than before.
• Business is driving the need for data. 
• Best practice delivery is hard in a fast changing environment. 
• Complexity is increasing in a diverse landscape.

Data of Christmas Yet to Come 

• The face of database administration is changing with multiple of types of engines, tools, applications and cloud offerings. 
•  It is necessary to have broad range of database knowledge to ensure best practice configurations are deployed for the plethora of tools.  
• Predicative analytics are becoming critical for business. 
• Deep learning utilizes machine learning to model data at high levels of abstraction which will transform how we live.
• Research is starting to become embedded in industry with the need to drive the next innovation.

SQLBits in Space

SQLBits XV was held between 4 -6 May 2016 at the Exhibition Centre in Liverpool. It was the official UK launch event of SQL Server 2016 which will RTM 1st June. There were lots of amazing sessions held for the first time in domes.

The keynote was delivered by Joseph Sirosh, the corporate vice president of the Microsoft Data Group. His keynote entitled the unreasonable effectiveness of data. A paper was written by Alon Halevy, Peter Norvig, and Fernando Pereira of the same title.  Joseph Sirosh mentioned the future effectiveness of data and the Sloan Digital Sky Survey, the 1st astronomy datascope. A take away was that there are many new data services and R should be the language to learn. 

There were many sessions covering the new features of SQL Server 2016 on both the BI and administration side. A highlight for me was the training day on data science by Buck Woody and using the Cortana Intelligence suite. 

The Cortana Analytics Suite big data and advances analytics process

The Azure IaaS and PaaS Services are embedded into these services.

  

U-SQL is another new language that allows you to query unstructured data. Michale Rys delivered a very informative session on Azure Data Lake and U-SQL. The traditional data warehousing approach is

  The new Data Lake approach

 The slides are http://www.slideshare.net/MichaelRys

 There are many new features in SQL Server 2016 and many new data features in Azure.

Microsoft Data Platform Specialist Certifications

There are a new set of certifications for Microsoft Azure

Microsoft Specialist: Cloud Data Platform Solutions

70-473 Designing and implementing Cloud Data Platform Solutions exam 

Microsoft Specialist: Big Data Analytics Solutions

70-475 Designing and Implementing Big Data Analytics Solutions exam 

More details:

http://buildazure.com/2016/03/09/microsoft-data-platform-specialist-certifications/

Microsoft's Vision for Data

Microsoft published a video about its vision for data.

How do you find certainty in a world where change is not just inevitable, it’s constant? When you have absolute faith in your data strategy, you can stop focusing on making the business run and start focusing on where you want your business to go.

Data Metaphors

The more articles you read the more you become aware of the use of metaphors in the phenomena of data descriptions. The comparisons which allow visualization of various images of unprecedented data growth seem endless

• Big Data  - http://www.capacent.is/library/Skrar/Analytics/economist-data-20100227.pdf
• Data Avalanche  - discussed http://kevinekline.com/2010/11/09/tn-summit-2010-surviving-the-data-avalanche
• Data Flood  - discussed in The Fourth Paradigm Data-Intensive Scientific Discovery http://research.microsoft.com/en-us/collaboration/fourthparadigm/
• Data Explosion   - The digital universe decade http://www.emc.com/collateral/demos/microsites/idc-digital-universe/iview.htm
• Data Revolution – from social data to open data sets http://giscience.org.uk/2011/03/21/the-open-data-revolution/
• Data Deluge  - http://www.sciencedaily.com/releases/2011/04/110422131123.htm
• Data Tsunami   - http://pubs.amstat.org/doi/pdfplus/10.1198/jasa.2011.ap10508