Welcome

Passionately curious about Data, Databases and Systems Complexity. Data is ubiquitous, the database universe is dichotomous (structured and unstructured), expanding and complex. Find my Database Research at SQLToolkit.co.uk . Microsoft Data Platform MVP

"The important thing is not to stop questioning. Curiosity has its own reason for existing" Einstein



Showing posts with label Data Science. Show all posts
Showing posts with label Data Science. Show all posts

Thursday, 2 April 2020

The Seattle Report on Data Research

I read with interest The Seattle Report on Data Research which presents the state of affairs for database systems. There are many interesting points raised about the diversity of the field and the challenges that are now apparent. These being:

  • Data Science having an emerging importance, 
  • HTAP (hybrid  transaction (OLTP) analytical processing (OLAP) creating real time operations to drive intelligent processing and reach, 
  • ethical responsibilities, 
  • scalable shareable data discovery 

This leads to the field having a bit of an identity crisis. We need to think about the end to end solutions that will provide insights whilst considering things such as data quality and data providence.

The question that we must all ask ourselves is, what is the vision for the future where data is all around us.


Sunday, 10 March 2019

Woman in Data Science (WiDS) Scotland


The Stanford University's Women In Data Science  (WiDS) initiative, The Data Lab, Turing's Testers along with their primary sponsor Mudano, have created an event that celebrates women, tech, innovation and codebreaking! This event brings together women data scientists and school girls to showcase what a data career looks like, and inspire the female data leaders of the future.  The aim is to inspire school girls to consider STEM and data related careers by bringing together the girls with inspiring women working in the field of data science and to expose them to some fun activities that are powered by data.  I have the privilege to attending the event, on behalf of my employer CGI, to participate as a mentor and to speak to the girls to share the wonders of working with data.

Women in Data Science is on 11 March 2019 at the National Museum of Scotland in Edinburgh. It is one of the fringe events of the UK’s first two week festival of Data Innovation in Scotland from the 11th to 22nd March 2019 and in its third year. DataFest will showcase Scotland's leading role in data science and artificial intelligence with networking from industry, academia and data enthusiasts. 

The Event details:

The Cyber Treasure Hunt has been created by Turing's Testers, a group of motivated pupils and STEM ambassadors; inspiring, engaging and supporting girls into the technology sector. This has been running over a number of months with codes being released every few weeks. For a school to gain invitation to this event they must crack the codes. This event will be the final code cracking session with the winner being announced at the end of the day.

The event will be broken into a number of sessions and vary between workshops and talks. Talks will be led by various female thought leaders from the world of tech, including none other than Hannah Fry.

Attendee spaces are limited at this event. We expect tickets to sell out quickly, however will have a waitlist and will inform you if you have secured a place. Attendees will have a hands on role as part of this event and will be asked to help out with mentoring and guidance to each of the groups throughout the day.

This event also plays part of DataFest, kicking off proceedings on the first day of the two week festival of data science.

Agenda;

10.00 - Registration
10.30 - First Rotation of Workshops and Talks
12.00 - Lunch
12.40 - Second Rotation of Workshops and Talks
14.10 - Prize Giving
15.00 - Closing Comments

Friday, 18 January 2019

Data Science Activities linked to Business

Managing Big Data, AI and Data Science all need new processes and methods to be efficient. I am always on the look out for new tools that help refine my thinking and usage.  William Schmarzo shared the Hypothesis Development Canvas as tool to connect data science to the organization. It is to be used to develop business hypothesis.


He also shared thinking like a data scientist process.























Sunday, 13 January 2019

Wednesday, 11 July 2018

Microsoft Research Open Data Sets

Microsoft Research Outreach team worked with the community to enable adoption of cloud based research. As a result they have launched  Microsoft Research Open Data,  a new data repository for the global research community. Microsoft wish to bring processing to the data rather than rely on data movement through the internet. This useful addition allows the data sets to be copied directly to the Azure based Data Science virtual machines. More details can be read here. The aim is to provide anonymized curated and meaningful datasets that are findable, accessible, interoperable and reusable. This follows on from the data-intensive science fourth paradigm of discovery discussed by Jim Gray.  

The open data set categories can be seen below. 


Saturday, 27 January 2018

Data Scientist Skills

There are an avalanche of skills required to become a data scientist. I came across this useful diagram.


The hierarchy of needs for data science could help you be more effective with AI and machine learning.




Wednesday, 17 January 2018

Field Guide to Data Science

I read this really useful  guide about data science. The Field Guide to Data Science was created to help organizations of all types and missions understand how to make use of data as a resource

More details about understanding the DNA of data can be found here.



Friday, 17 November 2017

Big Data LDN 2017


















I attended Big Data London 15-16 Nov 2017 with leading data & analytics experts showcasing their tools to help with delivering data-driven strategy. The conference showcased the fourth industrial revolution report which explains what the UK’s data leaders think about the state of the UK data economy.

A summary of things I found interesting during the two day event are summarized here.

Machine Learning is such a topical discussion point, but it is not that difficult to get started. An area to initially look at is co-occurrence and recommendation.  Co-occurrence helps you find behaviours and you can use that to find recommendations in areas such as textual analysis and intrusion detection.


Machine learning was described as the integration between analytics and operations. The three questions to ask were: what algorithm, what tools and what process. 90% of machine learning success is in data logistics (being able to handle lots of data types), not learning.

The CDO’s playbook was launched. The Chief Data Officer is a rapidly expanding role and this book offers practical advice on what this role is, how it fits into to other c-suite roles and provides actionable tips.

There are many challenges when dealing with citizen data. At the heart of audiences is

- single view of the customer
- deeper engagement
- supported intelligence
- relationship management

The main challenge is data quality and having a high enough quality of data to provide insight.

Citizens want to be data scientists and be able to dive into the data with ease. This self-service model can have challenges. Better governance, data management and operational efficiency are required together with the rise of managed service to remove the complexities of running these services.

The keynote on day 2, machine learning, AI and the future of big data analytics by Dr Amr 
Awadallah, Co-founder of Cloudera, talked about a history of waves.

- wave 1 automation of knowledge transfer
- wave 2 automation of food
- wave 3 automation of discovery
- wave 4 making and moving stuff (Industrial revolution)
- wave 5 automation of processes (IT revolution)
- wave 6 automation of decisions.

We are in wave 6 which is about collecting data and leveraging data to make decisions. It is different from the BI wave where humans made decisions. The new wave is learning how decisions are made and automating them. Things to consider for success are

- build a data driven culture
- develop the right team and skills
- be agile/lean in development
- leverage DevOps for production
- right size data governance 

There were discussions about data narrative and telling a story to the audience. The five steps learnt for better storytelling

- identify the right data
- choose the right visualizations
- calibrate visuals to your message
- remove unnecessary noise
- focus attention on what’s important

Matt Aslett talked on pervasive intelligence: the future of big data, machine learning and IoT, the details of which have been published in a report. He discussed trends and implications of the AI automation spectrum. It will bring about fundamental and wide ranging positive societal implication that will change the way we live, work, play, transact and travel. He mentioned a risk of having a small number of platform oriented companies that control the forces of production for generating value from data. The 4sight report on the future of IT is coming soon and sounds an interesting read.

Deep learning demystified explained why neural networks, that are not new, have only just come to the fore. It was because they were originally thought of as part of a failed experiment. In fact, it was that they did not use enough data. For supervised learning it works well with very large data sets. The key things to think of when considering deep learning are that it
- must have large data, a minimum of 10 million labels of data
- what level of accuracy do you need?
- can something simple work? – start with classical models such as linear models

There is a deep learning institute to learn more. 

The conference was useful and provided a wide range of discussions on high level data topics.

Saturday, 1 July 2017

Journey to the cloud

I have been reading a very interesting free ebook Enterprise Cloud Strategy by Barry Briggs and Eduardo Kassner. 

There are various types of modernisation discussed. 


It is based on Based on “Gartner Identifies Five Ways to Migrate Applications to the Cloud”, Gartner Inc., 2011. 

The overview of cloud migration principles discussed are summarised:























There is new a model that truly adopts cloud taking into account SaaS, PaaS and IaaS. This new architecture principle means considering the best placement of workloads, always looking at SaaS first.


This new way of thinking follows this placement decision tree which helps you make the correct decision on whether to use SaaS, PaaS, IaaS or private cloud.

On the data side, the paper talks about dividing your data into several categories related to risk. This is really about creating your own business data DNA. 'Data analytics' and 'BI and analytics'  cloud architectural blueprints are shared.

An interesting change for roles is the evolution of roles for the cloud. Microsoft suggest the data roles of Data Scientist and Information Architect emerge from the DBA and Statistician.










Monday, 1 May 2017

Interesting blog posts of the last month

These are a few blog posts I read last month which i found quiet interesting.

Azure SQL 
Resource Manager template reference now available
DEA 2.0 Technical preview: Release Overview – Database Experimentation Assistant: Overview
Azure Data Warehouse: Understanding the DWU, workload management, resource and memory allocation
Top 30 SQL Blogs  Websites for Microsoft SQL Server Administrators
Microsoft touts SQL Server 2017 as the first RDBMS with built-in AI

NoSQL
AWS Database Migration Service expands into NoSQL migrations with MongoDB & Amazon DynamoDB support

DocumentDB

Visualization

Getting started with R in SQL Server

Deep Learning

Graph
Build a recommendation system with the support for graph data in SQL Server 2017 and Azure SQL DB

Data

Book
The Data and Analytics Playbook: Proven Methods for Governed Data and Analytic Quality 1st Edition

Wednesday, 1 March 2017

Saturday, 18 February 2017

Thesis Defense


On Thursday I attended my viva examination at the Open University in Milton Keynes, for my doctorate, and was successful in defending my thesis. I was awarded the degree subject to amendments being made. I have six months to make these amendments which then have to be resubmitted for review before I finally gain my PhD.

My thesis is entitled: A Study into Best Practices and Procedures used in the Management of Database Systems.

















I gave a short presentation at the beginning of the viva, as an icebreaker, which shared a high level overview of the research. Following that the examiners asked many questions. The viva questions were hard, covering the end to end thesis and required a lot of careful  thinking about, to answer them succinctly and to explain the methods, literature and findings. 

I was exhausted at the end of the process but reflecting now feel I have been so privileged to have been able to be innovative and adding to the body of knowledge. It has been really exciting to be able to work jointly in industry and research. I strongly believe that with both industry and research working hand in hand, greater improvement and innovation can take place.

Sunday, 12 February 2017

Where is Data Going?

There are many data related visions at the current time.

McKinsey are saying "Big data’s potential just keeps growing. Taking full advantage means companies must incorporate analytics into their strategic vision and use it to make better, faster decisions.  A new report from the McKinsey Global Institute (MGI), The age of analytics: Competing in a data-driven world, suggests that the range of applications and opportunities has grown and will continue to expand"


Harvard Business Review are wrote an article discussing " "How Artificial Intelligence Will Redefine Management"  The fact is, artificial intelligence will soon be able to do the administrative tasks that consume much of managers’ time faster, better, and at a lower cost."








Microsoft discuss "Deep Learning is a hot buzzword of today... Deep Learning is now changing the very customer experience around many of Microsoft’s products, including HoloLens, Skype, Cortana, Office 365, Bing and more. Deep Learning is also a core part of Microsoft’s development platform offerings. The applications of Deep Learning technology are truly so far reaching that the new mantra, of Deep Learning in every software, may well become a reality within this decade...Furthermore, it’s not just about doing deep learning standalone, but it’s rather about combining it with all kinds of data and analytics "


A definition of deep learning could be said to be "a subfield of machine learning concerned with algorithms inspired by the structure and function of the brain called artificial neural networks."










In the operational world we should be dark data. Gartner describes dark data as "information assets that organisations collect, process and store in the course of their regular business activity, but generally fail to use for other purposes." This could lead to a revolutionary way to provide lean management of database systems which become fully automated.



















The research space is changing as Amazon, Google, Huawei, and Microsoft sponsor UC Berkeley RISELab, AMPLab’s successor. I think it will be interesting to see how MIT and Stanford, Berkeley go forward.

In the UK The Alan Turing Institute, was created by five founding universities – Cambridge, Edinburgh, Oxford, UCL and Warwick, and is the national institute for data science, headquartered at the British Library. Their mission is to make great leaps in data science research in order to change the world for the better. Data Science is the hot topic for Data and Business so it will be interesting to see how this institute evolves and changes how data science is incorporated into every business.

It seems to me that predictive analytics providing automation through all that dark data is a place requiring significant investment.