Big Data Storymap

This is really great diagram EMC have created setting out the current state and future state of BI and Big Data with the business challenges.

Cortana Analytics gives us some great tools to be able to assist with this storymap.

Thesis Defense

On Thursday I attended my viva examination at the Open University in Milton Keynes, for my doctorate, and was successful in defending my thesis. I was awarded the degree subject to amendments being made. I have six months to make these amendments which then have to be resubmitted for review before I finally gain my PhD.

My thesis is entitled: A Study into Best Practices and Procedures used in the Management of Database Systems.

I gave a short presentation at the beginning of the viva, as an icebreaker, which shared a high level overview of the research. Following that the examiners asked many questions. The viva questions were hard, covering the end to end thesis and required a lot of careful  thinking about, to answer them succinctly and to explain the methods, literature and findings. 

I was exhausted at the end of the process but reflecting now feel I have been so privileged to have been able to be innovative and adding to the body of knowledge. It has been really exciting to be able to work jointly in industry and research. I strongly believe that with both industry and research working hand in hand, greater improvement and innovation can take place.

Shift in Market Leaders

Gartner Magic Quadrant for Business Intelligence and Analytics Platforms reports the market shift in leaders

Data Warehousing and Modern BI technical pattern

There is a great article explaining the Data Warehousing and Modern BI technical pattern which is deployed via Cortana Intelligence Solutions. The artcile shows a hybrid EDW scenario and how it can be implemented on Azure.





















This model uses Azure SQL Data Warehouse, Azure Analysis Services,  Azure Blob Storage, 
Azure HDInsight and Azure Data Factory. More details can be found here.

Where is Data Going?

There are many data related visions at the current time.

McKinsey are saying "Big data’s potential just keeps growing. Taking full advantage means companies must incorporate analytics into their strategic vision and use it to make better, faster decisions.  A new report from the McKinsey Global Institute (MGI), The age of analytics: Competing in a data-driven world, suggests that the range of applications and opportunities has grown and will continue to expand"

Harvard Business Review are wrote an article discussing " "How Artificial Intelligence Will Redefine Management"  The fact is, artificial intelligence will soon be able to do the administrative tasks that consume much of managers’ time faster, better, and at a lower cost."

Microsoft discuss "Deep Learning is a hot buzzword of today... Deep Learning is now changing the very customer experience around many of Microsoft’s products, including HoloLens, Skype, Cortana, Office 365, Bing and more. Deep Learning is also a core part of Microsoft’s development platform offerings. The applications of Deep Learning technology are truly so far reaching that the new mantra, of Deep Learning in every software, may well become a reality within this decade...Furthermore, it’s not just about doing deep learning standalone, but it’s rather about combining it with all kinds of data and analytics "

A definition of deep learning could be said to be "a subfield of machine learning concerned with algorithms inspired by the structure and function of the brain called artificial neural networks."










In the operational world we should be dark data. Gartner describes dark data as "information assets that organisations collect, process and store in the course of their regular business activity, but generally fail to use for other purposes." This could lead to a revolutionary way to provide lean management of database systems which become fully automated.

The research space is changing as Amazon, Google, Huawei, and Microsoft sponsor UC Berkeley RISELab, AMPLab’s successor. I think it will be interesting to see how MIT and Stanford, Berkeley go forward.

In the UK The Alan Turing Institute, was created by five founding universities – Cambridge, Edinburgh, Oxford, UCL and Warwick, and is the national institute for data science, headquartered at the British Library. Their mission is to make great leaps in data science research in order to change the world for the better. Data Science is the hot topic for Data and Business so it will be interesting to see how this institute evolves and changes how data science is incorporated into every business.

It seems to me that predictive analytics providing automation through all that dark data is a place requiring significant investment.

Keeping Data Skills Up To Date in a Rapidly Changing World

Keeping up to date with rapid technological advances has become a real issue, especially with the adoption of cloud technologies. Gone are the days when traditional learning methods can be applied. It is possible that, if a group of people need to skill up on cloud data technology, and the training of people is staggered over several months that the first people are already out of date by the time the last person is trained.  Another issue is the prolific number of cloud technologies, tools, services and products that are changing at speed, so much so, that some products may only be around for one year before the consumer has to face a rapid migration process to a new solution.

To address these rapid changes it is necessary to adopt an adaptive learning style that is not only updated by exam every 3 years. It is necessary to have your own individual learning plan which has daily options, weekly options, monthly options and certification tracks. The key is to remain agile and build learning into your everyday routine.

My routine for keeping my specialism up to date involves

• Daily twitter reading and posting
• Blog post reading and writing
• Watching webcasts
• Free online courses
• Reading free ebooks 
• Reading Communications of the ACM 
• Reading of SIGMOD Record
• Attending conferences
• Attending free vendor days
• Practical practice using existing skills
• Practical practice of new skills
• Documenting things I learn
• Keeping certifications up to date

As part of that learning plan there has to be three elements: research, theory and practice.

There are always changes to vendor certifications to keep them relevant. The newer MSCE exam, Data Management and Analytics certification are to be taken each calendar year and you have the opportunity to earn the certification to receive an additional entry on your transcript. The Microsoft solution expert exams demonstrate your broad skill sets in SQL administration, building enterprise-scale data solutions, and leveraging business intelligence data—both on-premises and in cloud environments. Amazon do not have the same breadth of data certifications yet with the AWS certification being for Big Data only.

Two new Microsoft data exams:

70-473: Designing and Implementing Cloud Data Platform Solutions

70-475: Designing and Implementing Big Data Analytics Solutions

Amazon exams:

AWS Certified Big Data Certification – Specialty Beta Exams

Requires a current Associate level AWS Certified Solution Architect Certification