Maximize the value of your data with Azure Purview

There is a digital event coming on 28 September 2021 9-10am pacific time. The event is about achieving unified data governance with Azure Purview.
Join Microsoft Corporate Vice President Rohan Kumar at this free digital event for demos and deep dives.
Register now to:

• Learn to create a comprehensive, automated map of all your data.

• See how Azure Purview works with Azure Synapse Analytics, Power BI, and the rest of your data estate to deliver timely, reliable insights.

• Watch in-depth demos of product features including Azure Purview Data Map and Data Catalog.

• Ask Azure experts your data governance questions in the live Q&A.

 

Azure Purview August updates

There have been some exciting changes to Azure Purview announced. These changes relate to charging and permissions. 

Elastic data maps

The data map is the foundation for data discovery. The data map has two components the throughput created by CRUD operations and storage of the metadata.

The data map can now grow elastically starting at one capacity unit. A capacity unit includes a throughput of 25 operations/sec and the metadata storage scales in increments of 2 GB. Purview Data Map can automatically scale up and down within limits . This new charging model makes Purview much more user friendly and less costly to set up and run.

Access Control in Azure Purview

Note: only applies for Purview accounts created on or after 18 August 18 2021,

A collection is a tool to group assets, sources, and other artifacts into a hierarchy for discoverability and to manage access control. Collections are used to organise and manage assets.

The are various roles that exist:

• Collection admins - can edit Purview collections, their details, and add sub collections. They can also add users into other Purview roles on collections where they're admins.
• Data source admins - can manage data sources and data scans.
• Data curators - can create, read, modify, and delete catalog data assets and set up relationships between assets.
• Data readers - can access but not modify the data.

There is a great video to watch from W Strasser explaining this new data plane RBAC (role based access control) catalog permission. Being able to fine tune the access to the collections brings with it great advantages. Currently collection names can't be updated or deleted. 

A Summer Retrospective: a bygone era

A few weeks in rural France is just the place to contemplate life and take a step back into a bygone era. An era where there are no phones, no internet and no television. Life can quite easily pass you by and you could go for weeks not speaking to a sole. The fruit on the trees ripen in the orchard, the birds waiting for the perfect moment to swoop and eat the fruit. The roads are mostly empty with the occasional car or logging lorry passing by. Cycling is heaven with the roads to yourself.

This rural area, 214 million years ago, had all life within 300 miles of Rochechouart wiped out when a meteorite, around one of the 15 largest ever to come crashing down on earth. The geological signs of he creator are still present today. This bygone era was also rife with conflict. From the last battle of Richard the 1st - the Lionheart, who laid siege to the Chateau of Chalus-Chabrol, located at the border between Aquitaine and the French kingdom, to the hideouts in the forest of the Maquis du Limousin, who were one of the largest groups of French resistance fighters in the Second World War. The village of Oradour-sur-Glane remains an empty ruin as a memorial for the massacre of its inhabitants.

In this backdrop, technology seems a lifetime away. I can't stress enough the tremendous benefits of taking a technological break for your mental health. You can dream and innovative without the interruption of everyday life.

The age of cloud computing, big data and the algorithm requires a 360-degree perspective. A socio-technical perspective is critical. Reflecting on the changes to the earth, made to this unique landscape from space, you realize that data is in the environment. It is not possible to be an expert in all areas as data is the environment. Data is history, is in the maps, is used in conflict resolution and is used for impact analysis. Data is completely inseparable from life and it drives life, not only business. The  choice of tools available help you navigate through data are vast.

The question is can one truly ever master the entirety of life. Data is life, the past, the present and the future. To truly be a master it requires collaboration, communication and control, as data weaves its interconnected complexity throughout life. A holistic view of this diverse scientific area is required to provide a sustainable future. There is no one best practice that can help navigate this web of graph vertices and edges.

To that end I summise that taking a technological break enables the mind to contemplate and blue sky thinking roam free. Happy Summer break. 

Data Conferences

I have been surprised by the number of data conferences there are and the continuing growth and diversity of the topics and formats. As well as the main and specialist conferences, a few of which are listed below,  there are a huge number of training events that take place in the evenings and weekends such as  SQL Saturdays ,  Azure Data Community and Data Saturdays . With all these events providing learning opportunities ,and many for free, we are very lucky that we have a community that is so will to share their experience.

Conferences	Date	URL
Dativerse	13-Aug-21	https://datagrillen.com/dativerse/
Data Platform Virtual Summit	13-18 Sept 2021	https://dataplatformgeeks.com/dps2021/
Future Data Driven	29-Sep-21	https://datadrivencommunity.com/
DataMinds Connect	11-12 Oct 2021	https://datamindsconnect.be/
New Stars of Data	22-Oct-2021	https://www.newstarsofdata.com/
Data Weekender	06-Nov-2021	https://www.dataweekender.com/
PASS Data Community Summit	8-12 Nov 2021	https://passdatacommunitysummit.com/
The SQL Server & Azure SQL Conference	7-9 Dec 2021	https://www.mssqlconf.com/#!/
Data & AI Summit	27-30 June 2022	https://databricks.com/dataaisummit
Big Data LDN	22-23 Sept 2021	https://bigdataldn.com/
SQLBits	2022	https://sqlbits.com/
Azure Cosmos DB Conf	20-21 April 2021	https://gotcosmos.com/conf
Power BI Summit	7-11 March 2022	https://globalpowerbisummit.com/
Data Toboggan	12-Jun-2021	http://www.datatoboggan.co.uk/
Microsoft Build 2021	25-27 May 2021	https://mybuild.microsoft.com/home
Microsoft Ignite 2021	2-4 Nov 2021	https://myignite.microsoft.com/home
Microsoft Inspire 2021	14-Jul-2021	https://myinspire.microsoft.com/home
DataMinutes	22-Jan-2022	https://datagrillen.com/dataminutes/

Responsible Innovation: A Best Practices Toolkit

Responsible innovation is a toolkit that helps developers become good stewards for the future of science and its effect on society.  

There are 3 areas

• Judgment Call
• Harms Modelling
• Community Jury

This toolkit provides a set of practices currently in development, for anticipating and addressing the potential negative impacts of technology on people. This is an early release of this development.

Judgment Call 

Judgment Call is an award-winning game and team-based activity that puts Microsoft’s AI principles of fairness, privacy and security, reliability and safety, transparency, inclusion, and accountability into action. The game cultivates stakeholder empathy through scenario-imagining. Game participants write product reviews from the perspective of a particular stakeholder, describing what kind of impact and harms the technology could produce from their point of view.

To prepare for this game, download the printable Judgment Call game kit.

Harms Modelling 

Harms Modelling is a framework for product teams, grounded in four core pillars of responsible innovation, that examine how people's lives can be negatively impacted by technology: injuries, denial of consequential services, infringement on human rights, and erosion of democratic & societal structures. Similar to Security Threat Modelling, This modelling enables product teams to anticipate potential real-world impacts of technology.

Community Jury

Community Jury is a technique that brings together diverse stakeholders impacted by a technology. It is an adaptation of the citizen jury. The stakeholders are provided an opportunity to learn from experts about a project, deliberate together, and give feedback on use cases and product design. This responsible innovation technique allows project teams to collaborate with researchers to identify stakeholder values, and understand the perceptions and concerns of impacted stakeholders.

These 3  new tools under development are underdevelopment but quiet interesting to look at. 

References

Citizens Juries

The Ethics of AI Ethics: An Evaluation of Guidelines

Hagendorff, T. The Ethics of AI Ethics: An Evaluation of Guidelines. Minds & Machines 30, 99–120 (2020) 

Career paths in computing enabled me to

It is such a privilege to have an article published in the Communications of the ACM journal. 

July 2021, Vol. 64 No. 7, Page 7

DOI: 10.1145/3464919

It can be read here 

Data Governance: An Introduction

Initially published on the Coeo blog.  

Data Governance is a core area that businesses need to adopt in the data-driven world. Data has been around since the earliest of times, from the first libraries in the ancient world that started to collect and store information.

The collection of scientific research information, from census information about human populations, weather and spatial data to DNA genetic data, have all been contributing to the need to store data for analysis. The breadth of the information that is available for analysis covers our entire planet and beyond, and the population as well as different species. With our life and environment becoming documented to the finest degree the need for categorisation, data labelling and data management has become engrained into our society. Where research led the way for documentation of classification for data, business is now at a crucial time of growth and expansion to enable innovation.

With all data there becomes a continual need for its management and a core starting place is data governance. The DAMA Dictionary of Data Management defines Data Governance as “The exercise of authority, control and shared decision making (planning, monitoring and enforcement) over the management of data assets".

The goal of data governance is to help an organisation to manage data as an asset efficiently and effectively. It provides the principles, policy, processes, framework, metrics and oversight that are required to drive the most business value. Data governance programs have a goal of creating sustainable data management, good data quality that is measured and defining policies and practices. A much-needed area that needs to be considered is that of culture and embedding that culture of data management into the business.

We start with understanding what data assets a business has from the core known data and dark data; data that is collected but not used. The proliferation of duplicate data around a business is key to document. Often the first thing that comes to mind with data governance these days is compliance with all the data breaches that keep occurring. The areas one thinks of here are:  

• Policies
• Transparency
• Governance
• Regulations, such as GDPR
• Standards
• Rules
• Law

These require data inventories and audits to understand what personal data your organisation collects, where it is stored, how it is protected and who may have access to it.​ This is part of the picture that needs to be considered.

DAMA-DMBOK is an international guiding framework for the management of data. The framework includes areas such as:

• Data Strategy – defining, communicating and driving execution​.
• Policy – metadata management, access, usage, security, quality​
• Standards and quality – data architecture and data quality standards​
• Oversight/audit/stewardship​
• Compliance​
• Data issue management – compliance, ownership, policy, terminology, data quality, data access​
• Data management improvement projects ​
• Data asset valuation constantly define business value of data assets.

Consideration for the allocation of roles and responsibilities within an operating model helps guide the adoption of best practices.

In conclusion, managing data assets within a business requires it to be embedded in the culture of an organisation. Having high quality data leads to better business decisions. Having a core oversight function that is provided by a Chief Data Officer helps with keeping the day to day running of data in the fore front of everyone’s minds and you never know where the next innovation will come from.

More Information

• DAMA-DMBOK
• Non-Invasive Data Governance: The Path of Least Resistance and Greatest Success - Book by Robert S. Seiner A Guide to everything you need to know about dark data
• A brief history of big data everyone should read
• A very short history of big data