Enhancing Business Intelligence with Data Science

The heterogeneous nature of data has resulted in an evolution of the business intelligence platform. The traditional data warehouse architectures are now a part of a greater diverse set of products and tools available for use, to gain insight. This new architecture is in Microsoft Azure, which consists of information management, big data stores, machine learning and analytics and intelligence.

This huge number of tools are known as the Cortana Intelligence Suite. Cortana Intelligence is a platform and a process to perform advanced analytics from start to finish. It is a fully managed business intelligence, big data and advanced analytics offerings. Microsoft have been helping people learn the 14 new tools to explain and show how these fit together by using a mnemonic.

Say it Cortana, Cognitive Services, Bot Framework – intelligent assistant for speech and vision

See it Power BI – interactive report and visualization

Stream it Azure Stream Analytics – real time stream processing

Big it HD Insight – implementation of apache Hadoop

Learn it Azure Machine Learning and MRS – machine learning and R Server engine

Relate it Azure SQL DB, Data Warehouse, DocumentDB -  SQL and NoSQL engines

Store it Azure Data Lake –data storage and distributed processing

Bring it Azure Event Hubs – ingest data for web, IoT and apps

Move it Azure Data Factory – pipeline to move data in and out

Doc it Azure Data Catalog – documentation

Host it Microsoft Azure - IaaS, PaaS or SaaS

These tools are supplemented by a modified process model based on the CRISP-DM (Cross Industry Standard Process for Data Mining). CRISP-DM is a data mining process model that describes commonly used approaches that data mining experts use to tackle problems. CRISP-DM has six major phases. 

The Microsoft team science process is:

There are many tools to get started learning about data science and these a just a few.

A collection of data science tools

https://learnanalytics.microsoft.com/

Code samples

https://github.com/Azure/Cortana-Intelligence-Suite-Industry-Solution-How-To-Guides

https://github.com/Microsoft

Free eBooks from Microsoft Press - Microsoft Virtual Academy

https://mva.microsoft.com/ebooks

• Data Science with Microsoft SQL Server 2016
• Microsoft Azure Essentials: Fundamentals of Azure, Second Edition

Data science track in the Microsoft Professional Program

https://academy.microsoft.com/en-us/professional-program/data-science/

Data Intelligence

I had the amazing opportunity to attend PASS Summit 2016, the largest Microsoft SQL Server event I the world. The event provided the opportunity to meet many international experts and engage with Microsoft engineers in every field.

As a first time attendee there was a lot of logistics to understand to get the most from the event.  I was amazed by the number of Europeans who attended the conference, many of whom I know as a helper for many years at SQLBits. PASS Summit is the pinnacle of the year and I can say I gained much from this event which otherwise would not have been possible.

The first summit keynote delivered by Joseph Sirosh who presented types of A.C.I.D. intelligence with various patterns, intelligent DB, intelligent lake and deep intelligence. A.C.I.D. intelligence being Algorithms, Cloud, IoT and Data. Intelligence is now in every piece of software with applications that continually learn from the data and subsequent information.  This pushes intelligence to where the data lives.

The intelligent database incorporates the new functionality of R Services, provides an operating system of choice (Windows or Linux) for any data deployed anywhere.  The SQL Server 2016 functionality is extended with the hybrid transaction and analytical processing (HTAP) solution which the In-Memory OLTP, In-Memory Analytics, In-Memory Azure SQL Database (launched 15 November) combined with Polybase enable fast querying of structured and unstructured data. Polybase can connect to all data sources such as MongoDB, Hadoop, Teradata, Oracle.  Adding machine learning to the suite of tools add benefits such as real time fraud detection.  DocumentDB properties were also discussed highlighting the blazing fast performance and global replication.

The intelligence lake enables the handling of petabytes of data through algorithms and the extensible data lake. Azure analysis services is available at public preview and Azure SQL Data Warehouse with its parallel processing and scale out was offered as an exclusive one month free trial. There was a great demo by Julie Koesmarno on Azure cognitive services with U-SQL which provided sentiment analysis of War and Peace.

The final part of the key note presented deep learning which looked at many real life examples of learning everywhere from collecting data reviewing whether power lines looked in a good state of repair to face detection to medical research detecting cancer cells.

The keynote was truly inspirational. There were many other amazing sessions with a vast amount of information on diverse topics which I will share in separate posts.

Innovation and Knowledge Development Research Award

Friday 7 October was a very special day for me. I attended the Annual Association of Open University Graduates Research Awards Ceremony at The Open University in Milton Keynes with my family. I was amazingly privileged to be the recipient of the 2016 AOUG Will Swann Award for Innovation and Knowledge Development for dedication and outstanding achievement in postgraduate research.

This followed the submission of my thesis A Study into Best Practices and Procedures used in the Management of Database Systems for which I am awaiting my viva voce exam. 

Microsoft Machine Learning and Data Science Summit 2016

The Machine Learning & Data Science Summit took place in Atlanta between 26 - 27 September 2016.

There are many Videos from the Microsoft Machine Learning & Data Science Summit in Atlanta to watch.














I have so far watched the great Day 2 Keynote Session by Dr. Edward Tufte on The Future of Data Analysis included 

"Data analysis seeks to learn from experience.  Better inferences require better thinking and better tools. Practical advice about how to   make more credible conclusions based on data. What we can expect in the future, and what we should aspire to in the future. "

The many other videos cover many topics including practical patterns to jump start your analytic solutions to data lake patterns and practices.

Microsoft Streamlines Technical Certifications

Microsoft has announced a streamlining of certifications.

The Born to Learn site contains the full details,

The five new expert certifications are:

• MCSE: Cloud Platform and Infrastructure - focusing on skills validation for Windows Server and Microsoft Azure

• MCSE: Mobility - focusing on skills validation for Windows Client and Enterprise Mobility Suite

• MCSE: Data Management and Analysis - focusing on skills validation for both on-premises and cloud-based Microsoft data products and services

• MCSE: Productivity - focusing on skills validation for Office 365, SharePoint, Exchange, and Skype for Business

• MCSD: App Builder - focusing on skills validation for Web and Mobile app development

To earn each of these credentials, you must first earn a qualifying Microsoft Certified Solutions Associate (MCSA) certification and, then, pass a single additional exam from a list of electives associated with the corresponding Center of Excellence.   

Every year, you need to take an additional exam from the list of electives, demonstrating your investment in broadening or deepening your skills in a given Center of Excellence to retain the certification.  Each time you earn the certification, a new certification entry will be added to your transcript. 

Thesis Submission

After 6 years working on my part time PhD entitled A Study into Best Practices and Procedures used in the Management of Database Systems, it is complete. I wrote about 95,400 words.















The final package was posted for examination.

DocumentDB Tools

There is a useful cheat sheet to help with writing DocumentDB queries. This can be downloaded here

Estimate Request Units and Data Storage
https://www.documentdb.com/capacityplanner

Query Playground converts T-SQL Syntax to JSON
https://www.documentdb.com/sql/demo