A Guide to Complexity of Database Systems

The Phd research I undertook examined the complexity of database systems. A summary of the findings are provided. 

In the turbulent fast moving field of database systems, complexity is found everywhere. The volume, variety and velocity of data is continually expanding as well as the accessibility and realization that businesses have a wealth of untapped data that can be democratised. Not only this, and changes in new technology, but also with the shift in business markets, organisational changes, knowledge required by operating staff and numerous stakeholders, adds to the complexity. Many of these complexities have been discussed in the Claremont and Beckman Reports (Agrawal et al. 2009; Abadi et al. 2016). 

This guide takes a 360 degree view of the situation through a systems thinking lens, providing synthesis between the cross disciplinary fields. To be able to explain what complexity is shapes our understanding of the situation and a basic visualisation of this is shared through the use of a graph. The usage of graphs as visual representation are discussed with the presentation of the graph metrics leading to the CODEX, a blueprint for the management of database systems. The CODEX could enable transformation of the management of database systems so that actionable insight can be achieved.

Data Story Telling Booklet and Video

Story telling is an important facet when working with data. Whilst I was working on my research PhD, I thought it was important to share the main stages of my research.  The storyboard is shared as two resources. These are:

A PhD storyboard booklet
A collated storyboard video

More research articles see http://sqltoolkit.co.uk/

Microsoft SQL Server Machine Learning

I am pleased to be mentioned in a TechTarget article.

Innovative Designs for Innovative Thinking

These Spheres in Seattle will be home of a botanical garden, waterfalls, a river and tree-house like. They are disruptive, pioneering and futuristic to let Amazonians break free for innovative thinking. What an amazingly cool idea. 

PASS Summit 2017 Day 2 Keynote

I attended the Day 2 Keynote at PASS Summit presented by Rimma Nehme on Globally Distributed Databases Made Simple. This was an amazing presentation. It was presented seamlessly, explaining the technicalities of CosmosDB and how the globally distributed database works from the ground up.

Rimma raised the question, do we need another database? Databases need to meet the data needs for today and the future. Data is global, with large volumes of data being created every 60 seconds, which are continually growing and data is interconnected. The balance is shifting in the type of data and we need to have data globally next to users for processing, meaning the architecture needs to be different. 

CosmosDB was originally call Project Florence and was named as such because it is the place where the renaissance began. It was built in the cloud database for global distribution, with a fully resource governed stack and schema agnostic service. A single system image is used for all globally distributed resources. 

The resource model may have a database account / database that may span clusters and regions. The database is scaled out in terms of containers. It is designed to scale throughput and storage independently. There are two parts to the design. The physical system design is:

The partitioning system design is:

The design is to enable elastically scalable storage, throughput, anywhere, anytime.

Resource governance cannot be an afterthought. The request unit/sec (RU) is the normalized currency.

There are 5 well-defined consistency models in Azure Cosmos DB  with clear trade offs: strong; bounded-stateless; sessions; consistent prefix and eventual.

There is native support for multiple data models with more coming in the future.

The talk continued to cover how indexing works in depth and the key points to remember about Cosmosdb are:

The talk concluded with a great quote “It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is most adaptable to change.”

The slides can be downloaded.

PASS Summit 2017 Day 1 Keynote

I attended PASS Summit 2017 which was my second year of attendance. I enjoyed the conference enormously. It is enjoyable being immersed in data and being with people who are enthusiastic in the field.

The Day 1 Keynote "Microsoft for the Modern Data Estate" was presented by Rohan Kumar. Data is driving transformation. Data, Cloud and AI are the three most disruptive trends of our time. The modern data estate, enables simplicity and common sense. It takes any data from any source, structured or unstructured data and large or small data. The modern data estate provides a seamless infrastructure between on premises, private and public cloud, enabling a hybrid set up that hides the dichotomy of these disparate systems. Seamless flexibly and a choice of engines.

New features in SQL Server 2017

There are many changes to SQL Server 2017. SQL Server 2017 has industry leading performance and security now on Linux and Docker. The key engine changes 

• Support for graph data and queries
• Advanced Machine Learning with R and Python
• Native T-SQL scoring
• Adaptive Query Processing and Automation Plan Correction

SQL Server 2017 will enable deployment in seconds on Linux and Windows containers and has special pricing for SQL Server on Linux and Red Hat Enterprise Linux.

New Features Azure SQL Database

Azure SQL Database offers intelligent DBaaS, privacy and trust, seamless and compatibility and competitive TCO. There is seamless migration to the cloud with the cloud first approach breading faster innovations. The list of changes presented

Azure Data Factory now provides a managed environment for SQL Server Integration Services (SSIS) packages and easily move your SSIS workloads to cloud.

There was the announcement made for a new tool called, Microsoft SQL Operations Studio, a free lightweight modern data operations tools for SQL everywhere.

These are but some of the changes coming to the products.

Machines that learn to see and move: The future of artificial intelligence

I attended the Institute for Mathematical Innovation (IMI) public lecture by Professor Andrew Blake, Research Director at The Alan Turing Institute on 18 October. Professor  Blake is a pioneer in the development of algorithms that make it possible for computers to behave as seeing machines. Before joining the Institute in 2015, Professor Blake held the position of Microsoft Distinguished Scientist and Laboratory Director at the Microsoft Research Lab in Cambridge, and he has been on the faculty at Oxford University. He is a part of a new startup FiveAI.

The session abstract:

Neural networks have taken the world of computing in general and artificial intelligence (AI) in particular by storm.

But in the future, AI will need to revisit these generative models which are used to make predictions. There are several reasons for this – system robustness, precision issues, transparency, and the high cost of labelling data.

This is particularly true for perceptual AI, needed for autonomous vehicles, where the need for simulators and the need to confront novel situations, will demand further development of generative, probabilistic models. 

He talked about the empirical detector and generative model. At the moment it is the era of deep learning and neural networks, that sit within the empirical detector area. A black box area of big data and optimal predictive power. The generative model is analysis by synthesis and comes with an ‘explanation’, like a model. It starts with a hypothesis, typically probabilistic. Professor Blake believes the generative model will come back as perceptual models need this. This is

• to simulate labelled data
• for data fusion - to increase reliability
• to make detailed interpretations 
• for online simulation - to explain hard to read situations

This was a very insightful lecture and very interesting to see the mention of analysis by synthesis.