A Summer Retrospective: a bygone era

A few weeks in rural France is just the place to contemplate life and take a step back into a bygone era. An era where there are no phones, no internet and no television. Life can quite easily pass you by and you could go for weeks not speaking to a sole. The fruit on the trees ripen in the orchard, the birds waiting for the perfect moment to swoop and eat the fruit. The roads are mostly empty with the occasional car or logging lorry passing by. Cycling is heaven with the roads to yourself.

This rural area, 214 million years ago, had all life within 300 miles of Rochechouart wiped out when a meteorite, around one of the 15 largest ever to come crashing down on earth. The geological signs of he creator are still present today. This bygone era was also rife with conflict. From the last battle of Richard the 1st - the Lionheart, who laid siege to the Chateau of Chalus-Chabrol, located at the border between Aquitaine and the French kingdom, to the hideouts in the forest of the Maquis du Limousin, who were one of the largest groups of French resistance fighters in the Second World War. The village of Oradour-sur-Glane remains an empty ruin as a memorial for the massacre of its inhabitants.

In this backdrop, technology seems a lifetime away. I can't stress enough the tremendous benefits of taking a technological break for your mental health. You can dream and innovative without the interruption of everyday life.

The age of cloud computing, big data and the algorithm requires a 360-degree perspective. A socio-technical perspective is critical. Reflecting on the changes to the earth, made to this unique landscape from space, you realize that data is in the environment. It is not possible to be an expert in all areas as data is the environment. Data is history, is in the maps, is used in conflict resolution and is used for impact analysis. Data is completely inseparable from life and it drives life, not only business. The  choice of tools available help you navigate through data are vast.

The question is can one truly ever master the entirety of life. Data is life, the past, the present and the future. To truly be a master it requires collaboration, communication and control, as data weaves its interconnected complexity throughout life. A holistic view of this diverse scientific area is required to provide a sustainable future. There is no one best practice that can help navigate this web of graph vertices and edges.

To that end I summise that taking a technological break enables the mind to contemplate and blue sky thinking roam free. Happy Summer break. 

The Mirage and Metamorphosis of Data and AI

I have just taken a break to rejuvenate my creative juices. It was a time to reflect and innovate. We are often so busy in our day to day lives we don't stop and reflect. I spent my time reading and catching up on bleeding edge technology. I am always fascinated to see what is coming next, what problems researchers are trying to address and how Data and AI could be utilised to benefit industry and the world around us.

The role I enjoy the most is as a Data and AI philosopher providing thought leadership. We are at an exciting time in history to witness and contribute to the mirage and metamorphosis of Data and AI. My explorations find exciting challenges in diversity and Data and AI at the centre of most things we want to achieve. Research is increasingly needed in industry to achieve business success due to the increasing complexity within industry and the world around us. We need to move away from agile for certain tasks to enable complexity to be understood and use systems thinking techniques.  My findings on the future mirage and metamorphosis of Data and AI are a complex interconnected world around Data and AI and mastering that complexity is the key to success.

  

References
The data and AI market landscape 2019: The next wave of hybrid emerges
https://www.zdnet.com/article/the-data-and-ai-market-landscape-2019-the-next-wave-of-hybrid-emerges/
Part I: A Turbulent Year: The 2019 Data & AI Landscape
https://mattturck.com/data2019/
Part II: Major Trends in the 2019 Data & AI Landscape
https://mattturck.com/2019trends/
Navigating AI hype in search of success, Oliver Pickup (Sunday Times 12 May 2019)
The real big-data problem and why only machine learning can fix it
https://siliconangle.com/2019/08/09/real-big-data-problem-machine-learning-can-fix-mitcdoiq-startupoftheweek/
Big Data is just Data
https://buckwoody.wordpress.com/2019/08/26/big-data-is-just-data/
Maximising the AI opportunity
https://info.microsoft.com/rs/157-GQE-382/images/UK-DIGTRNS-CNTNT-content-MGC0003240.pdf
The Data Ethics Framework principles
https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework

Research skills for industry experts

It is great to see the name of the SQL Relay conference change to Data Relay. The conference encompasses the full breath of the Microsoft data platform and provides free Microsoft Data, AI & Analytics training conferences on your doorstep.

I am privileged to be speaking on Friday at Data Relay in Bristol to share my experience of providing high quality data analytics for my research using the Microsoft Data Platform. 

Understanding Complexity with Systems

I have today been watching the many butterflies flapping their wings around a Buddleia bush. The butterfly bush with its abundant deep violet-purple flowers attracts the butterflies, which surround the bush. I always wonder if this activity will cause a transformation round the globe with the much discussed 'butterfly effect'. The classic quote 'the notion that a butterfly stirring the air today in Peking can transform storm systems next month in New York [...] tiny differences in input could quickly become overwhelming differing in output' Glick (1987) . This is known as Chaos,  the term coined by Lorenz for a system with unpredictable outcomes.  The problem is that it is difficult to know the exact starting point of a situation accurately enough to put it into a mathematical formula. Thus each step in the process in the system moves further away from where you thought it should go. The errors or uncertainty multiply and can cause turbulent features. 

The natural world is unpredictable and has many patterns of behaviour. To understand any system well enough it is necessary to understand the inputs and outputs. To understand database systems there is much complexity that needs to be considered. This graph shows some of the theoretical components.

To apply this understanding and advance a system using artificial  intelligence (AI) you need to fully understand the system under investigation. For data and database systems people gain that experience and learning over many years. The art is to document the inputs and outputs and identify the best practices which have worked and those that have not. Also to create a method to connect the continuously evolving and changing best practice. Only then can you identify the opportunities within the system to improve management and create a state where AI is embedded within a helpful tool that could improve efficiency and performance.

My research examined the complexity of managing database systems and as such has the building blocks to begin to build an autonomous AI system to help manage database systems. Below shows the components that are interconnected in the the management of database systems.

Complexity is always changing and migrating with the passage of time and the trend will be from order to disorder, thus creating an autonomous system to help prevent that could be beneficial.The aim would be to create a self organizing system based on feedback from a persons behaviour, decisions, documentation, operational configuration, meetings and actions. 

CODEX: The Control of Data EXpediently

Complexity exists in database systems whether managing data or databases. The ability to understand this complexity and use this understanding to improve and innovate in the management of these database systems is an important step.  My PhD research investigations centred around understanding best practices and the complexity that exists around management of database systems. 

I used the analogy of a CODEX. The CODEX is a blueprint for database systems management. The acronym CODEX was selected by analogy with the revolutionary introduction of the Codex (Netz & Noel 2007, pp.69–85) in the first century AD which changed the storage medium from a roll to a Codex (book format). This brought challenges migrating the data, but significant benefits of increased speed of data access, reference and durability (of the parchment). Not all texts were migrated from rolls to Codex and those that were not migrated became defunct. Text case was changed from capitals to lowercase and minuscule copies made, resulting in further change; original majuscule manuscripts have not survived. This scholarly activity led to a revival in reading classic documents and a development of a centre of culture.

The CODEX is one of the outputs from the research, based on interpretation of the data. The components stated in the CODEX, are the most prevalent components that are connected when managing database systems.

The CODEX (Control of Data EXpediently) blueprint acronym is constructed as: C for control; O for control of Operations; D for data; E for expediently; and X for unpredictable events. It is an acronym for a system or way of controlling operations and data in a rapid, efficient and accurate manner.

The five inputs into the CODEX are required for every piece of data or database management work. These five inputs are described in the paragraphs below.

C. An important step in any database system is the control system: defining the business needs, budget, controlling the people and time factors. People are important in the management of the database system. It involves the stakeholders and the teams working together to achieve a single goal. The culture driving this collaborative venture forward will undoubtedly raise conflict, but this should be integrated with a high level of communication with all levels in management, the stakeholders, the teams and data and database staff. Also, the governance related to data and data quality should be controlled.

O. Control of operations of the database system is the core day to day running of management tasks, the processes and the performance of the system, orchestrating management through automated and self-managing systems where possible. Technical management needs to understand how internal and external technologies integrate. This is vital when using cloud technologies because internal managers have no control of the details. All of these operations require security to be considered to protect the data.

D. The increasing volume of data acquired today requires storage in various forms. Thus, databases or big data solutions have developed to satisfy the current demand not only for storage but also to provide information quickly and accurately. The variety of data, big or small, requires governance and has a purpose. The reporting and visualization of data is key to enhance business ability to grow, adapt and understand the complexity. Data is continually changing and more of it needs to be stored to meet the demands of society. Being able to understand the data for it to be available and useful, is a core requirement to improve and innovate.

E. Expediency is driven from the need to have efficient control over costs, speed of delivery and change. Designing database systems that are easy to manage, simple and agile utilising reference architectures and blueprints is key to performing expediently. To be able to proceed the critical factors are knowledge, skills, learning, leading to understanding and allowing planning to unfold unhindered. Development can lead to fast performing applications and efficient management through automation.

X. With any system and particularly in a diverse and ubiquitous database system, systems change is always happening, be it with the number and type of database platforms, the new technologies, global business or environment change. Change is rapid and diverse. Using patterns and always establishing best practice will help in the management of database systems. These best practices need to be able to rapidly change as requirements change or are not known at the outset. Producing documentation that can be automatically created is key for accuracy and ensuring documentation is available. Also, unpredictable events can occur and any changes to the components must be documented and changes to all respective components made. There is continuous feedback over time. 

I will discuss in another blog post, how the suggested pattern, CODEX (Control of Data EXpediently), can help improvement in managing database systems. Using AI to improve management, incorporating telemetry, and systems diagramming will aid with the change to come.

Holt, Victoria (2017). A Study into Best Practices and Procedures used in the Management of Database Systems. PhD thesis The Open University

Deep learning

Deep Learning is a subset of machine learning which aims to solve thought related problems. To understand this bleeding edge technology here are a few links.

Deep learning explained

Learn an intuitive approach to building the complex models that help machines solve real-world problems with human-like intelligence with the Microsoft AI school.

Free webinar

On 31 May there is a free webinar with Jen Stirrup on Deep Learning and Artificial Intelligence in the Workplace . The webinar asks what is Microsoft’s approach to Deep Learning, and how does it differ from Open Source alternatives? In this session, it will will look at Deep Learning, and how it can be implemented in Microsoft and Azure technologies with the Cognitive Toolkit, Tensorflow in Azure and CaffeOnSpark on AzureHDInsight

How deep learning will change customer experience

This article discusses artificial neural networks for machines which will allow machines and devices to function in some ways as humans do.

Continual Change and Complexity

This year has been an entire year of change for me, that will continue into the new year.  Continual change is the way of the new world. With data and AI being embedded into every realm of technology, we can expect more frequent and smaller changes on a day to day basis. I have enjoyed researching immensely and being able to apply that research to understanding the complexity of real world database problems.

As the holidays approach I wish you all a very Merry Christmas and Happy a New Year.

A Guide to Complexity of Database Systems

The Phd research I undertook examined the complexity of database systems. A summary of the findings are provided. 

In the turbulent fast moving field of database systems, complexity is found everywhere. The volume, variety and velocity of data is continually expanding as well as the accessibility and realization that businesses have a wealth of untapped data that can be democratised. Not only this, and changes in new technology, but also with the shift in business markets, organisational changes, knowledge required by operating staff and numerous stakeholders, adds to the complexity. Many of these complexities have been discussed in the Claremont and Beckman Reports (Agrawal et al. 2009; Abadi et al. 2016). 

This guide takes a 360 degree view of the situation through a systems thinking lens, providing synthesis between the cross disciplinary fields. To be able to explain what complexity is shapes our understanding of the situation and a basic visualisation of this is shared through the use of a graph. The usage of graphs as visual representation are discussed with the presentation of the graph metrics leading to the CODEX, a blueprint for the management of database systems. The CODEX could enable transformation of the management of database systems so that actionable insight can be achieved.

Doctor of Philosophy: Research Synopsis

I am pleased to announce that after 7 years of commitment to the field and effectively doing 2 full time jobs I met the academic requirements for the Doctor of Philosophy with The Open University. It has been an amazing few years researching something one finds fascinating.

The research synopsis is below.

There has been a vast expansion of data usage in recent years. The requirements of database systems to provide a variety of information has resulted in many more types of database engines and approaches. A once simple management task has become much more complex. Challenges exist for database managers to make the best choices of practices and procedures to satisfy the requirements of organisations. 

This research was aimed at understanding how the management of database systems is undertaken, how best practices and procedures form a part of the management process, and the complex nature of database systems. The study examined the adoption of best practices and how the complex interactions between components of the database system affect management and performance.

As part of this research, an innovative method was developed in which thematic analysis of the resulting data was deepened through the use of systems thinking and diagramming. Taking this holistic approach to database systems enabled a different understanding of best practices and the complexity of database systems. A ‘blueprint’, called a CODEX, was drawn up to support improvement and innovation of database systems. Based on a comprehensive assessment of the individual causal interactions between data components, a data map detailed the complex interactions.

My publications are on Google Scholar. 

Research meets Industry

Academic research and industry have often had siloed approaches. Academic research is essential to drive an industrial innovation. Often this research takes many years and may or may not result in commercial application. Over the last few years universities have been changing their old mind set and presentation, trying to raise the profile of their research and create impact in the community. It is about telling the story of the research and how that impacts society, that is key.

Large organisations in industry are beginning again to have their own research departments. In the last decade we saw the closure of many industry research laboratories as they were seen as a luxury and not fully integrated with businesses. Academic research has often not addressed the individual needs of industry and has, in some cases, not been keeping up with the speed of change in industry. There is a revival of organisations creating research facilities or investing in research and development. Those smaller to medium companies which had research divisions, are also revamping their presence. Competitive advantage can be obtained through the results of innovation.

Artificial Intelligence (AI) brings new challenges to research. Collaboration between big players, such as Microsoft Cortana and Amazon Alexa, aim to drive innovation forward. Once AI starts to be integrated into products and businesses, the insights and intelligence it brings, will begin to shape a new way of creating commercial applications and possibilities for innovation.

The world today is full of complexity and data is the fuel that helps drive intelligent action. Using data science, it helps breach that multi-disciplinary area to benefit all. Businesses must be grounded in accurate data to produce the next industrial revolution, through innovation and AI.

There is a new computational research centre of excellence (CoSeC) in the UK. The centre is bringing together leading UK expertise in key fields of computational research to tackle large-scale scientific software development, maintenance and distribution. This will improve scientific research software, that is relied upon worldwide, by universities and industries.

Thus, having a clear agenda that combines research and industry is the route to successfully navigating the fast moving business landscape.

Articles

So long to the silos 

Breaking Down Silos In Academic Cancer Research 

Systemsnot silos: new industrial strategy must deliver on a bold, long-term vision

The rise and fall of corporate R and D Out of the dusty labs

New R and D Centres: Major funding for new crop science research centre that will be ‘centrepiece‘ of industrial collaboration 

Water Innovation and Research Centre WIRC @Bath 

AlanTuring Institute up and running

ONS looks to big data as it explores new ways to measure UK economy

Microsoft  Research AI (MSR AI)

‘Cortana, Open Alexa,’ Amazon Says. And Microsoft Agrees
New computational research centre of excellence

Inspire Opening Keynote

I watched the Inspire keynote yesterday. It started with  @MelissaArnot (the first American woman to summit Everest without supplemental oxygen) talking about following her dreams with determination and collaboration. Dreams are a global language. Her introductory speech was so inspiring, encouraging you to never give up and to press on amidst adversity, to reach your goal.

Microsoft talked about the Intelligent cloud and intelligent edge. Technological changes are no longer bound to a single device. There is a profound shift to include AI in every application that is built. Data has gravity. It is about managing complexity on a new efficient frontier.

The new changes will be more profound that virtualisation. The new pillars for enabling digital transformation are: the modern workplace, business applications, applications & infrastructure and data & AI. It is about democratisation and making use of data and service, which are now digital. It is important to incorporate the customers needs in the solutions that we create.

Microsoft introduced a new intelligent solution, Microsoft 365. The modern workplace is shaping culture and it is necessary to empower people, to help share and transform, in disparate teams. It needs to be one simple framework or integrated system.

There were various other announcements but it is Data & AI changing model, that excites me.  Serverless and Microservices will be a revolutionary change but it is the fact that AI will be built into all services, that will define the next era.   Incorporating database richness, connecting into one digital data estate is the future, although it is not just about big data and analytics. It is about now trying to create value with small data.

Without data there is no AI. Data is the foundation.

It is not possible to succeed alone, so choosing your Partners is crucial. 

Database Lifecycle Management

This survey report released discusses the expertise required for managing data complexity. The findings identify the need for database expertise to manage this challenging environment.