Leverage data for building

The leverage data to build intelligent apps presentation gives an insightful overview of the Microsoft Data Platform and how to innovate with analytics and AI. 

PASS Summit 2017 Day 2 Keynote

I attended the Day 2 Keynote at PASS Summit presented by Rimma Nehme on Globally Distributed Databases Made Simple. This was an amazing presentation. It was presented seamlessly, explaining the technicalities of CosmosDB and how the globally distributed database works from the ground up.

Rimma raised the question, do we need another database? Databases need to meet the data needs for today and the future. Data is global, with large volumes of data being created every 60 seconds, which are continually growing and data is interconnected. The balance is shifting in the type of data and we need to have data globally next to users for processing, meaning the architecture needs to be different. 

CosmosDB was originally call Project Florence and was named as such because it is the place where the renaissance began. It was built in the cloud database for global distribution, with a fully resource governed stack and schema agnostic service. A single system image is used for all globally distributed resources. 

The resource model may have a database account / database that may span clusters and regions. The database is scaled out in terms of containers. It is designed to scale throughput and storage independently. There are two parts to the design. The physical system design is:

The partitioning system design is:

The design is to enable elastically scalable storage, throughput, anywhere, anytime.

Resource governance cannot be an afterthought. The request unit/sec (RU) is the normalized currency.

There are 5 well-defined consistency models in Azure Cosmos DB  with clear trade offs: strong; bounded-stateless; sessions; consistent prefix and eventual.

There is native support for multiple data models with more coming in the future.

The talk continued to cover how indexing works in depth and the key points to remember about Cosmosdb are:

The talk concluded with a great quote “It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is most adaptable to change.”

The slides can be downloaded.

PASS Summit 2017 Day 1 Keynote

I attended PASS Summit 2017 which was my second year of attendance. I enjoyed the conference enormously. It is enjoyable being immersed in data and being with people who are enthusiastic in the field.

The Day 1 Keynote "Microsoft for the Modern Data Estate" was presented by Rohan Kumar. Data is driving transformation. Data, Cloud and AI are the three most disruptive trends of our time. The modern data estate, enables simplicity and common sense. It takes any data from any source, structured or unstructured data and large or small data. The modern data estate provides a seamless infrastructure between on premises, private and public cloud, enabling a hybrid set up that hides the dichotomy of these disparate systems. Seamless flexibly and a choice of engines.

New features in SQL Server 2017

There are many changes to SQL Server 2017. SQL Server 2017 has industry leading performance and security now on Linux and Docker. The key engine changes 

• Support for graph data and queries
• Advanced Machine Learning with R and Python
• Native T-SQL scoring
• Adaptive Query Processing and Automation Plan Correction

SQL Server 2017 will enable deployment in seconds on Linux and Windows containers and has special pricing for SQL Server on Linux and Red Hat Enterprise Linux.

New Features Azure SQL Database

Azure SQL Database offers intelligent DBaaS, privacy and trust, seamless and compatibility and competitive TCO. There is seamless migration to the cloud with the cloud first approach breading faster innovations. The list of changes presented

Azure Data Factory now provides a managed environment for SQL Server Integration Services (SSIS) packages and easily move your SSIS workloads to cloud.

There was the announcement made for a new tool called, Microsoft SQL Operations Studio, a free lightweight modern data operations tools for SQL everywhere.

These are but some of the changes coming to the products.

Azure SQLDB Managed Instance and Database Migration Service

There are 2 services in preview at the moment Azure SQLDB Managed Instance and Database Migration Service. Both these services seem very useful for moving to the cloud.

They enable migration of a fleet of SQL Server databases without application re-architecture to the cloud.

Azure SQLDB Managed Instance provides security isolation with Azure Virtual Network, along with built-in HADR, built-in intelligent performance tuning, and intelligent security services that are already available in Azure SQL Database. No public endpoint is required.

Databases up to 35TB can be migrated.

It can be set up with a simple restore of native databases or log shipping using Azure Database Migration Service

Azure SQLDB Managed Instance provides SQL Server application compatibility including items such as commonly used cross database references using three-part names, CLR and R Services, DB Mail, SQL Agent, Transactional Replication, Change Data Capture and Service Broker.

State of the SQL Nation and the Microsoft Engineering Model

SQLBits 2017 was different this year and instead of a keynote launching the event they had a 15 mins Q & A session with Conor Cunningham and Simon Sabin. They discussed the rapid change of new engineering features in the product. There are now new features shipped monthly. The new model requires running test 24/7 and the interesting change is that all builds have the same level of sign off, increasing the quality of the build now. This is why the quality of the 2 monthly cumulative updates has increased. They run over 700,000 functional automated tests and stress test as well. They aim to ship with the minimum viable product and release updates every 3 to 6 months on them. They don’t deprecate features specifically unless there is a security risk and they use telemetry to help measure feature adoption.

SQL Server / Azure Engineering Model

A second session I attended was about the completely new engineering model that Microsoft now operate. There were some very interesting changes that I think all businesses need to think about when designing and structuring new teams to adapt to the changing model.

Microsoft DevOps Interesting Facts

• Microsoft have around 1.7 million production databases.
• They have no Database Administrators (DBAs).
• They have no Operations team.
• Engineers automate everything.
• SQL Server Azure runs 100% DevOps.
• SQL Server is managed by telemetry. 600TB of logs are collected every day and machine learning algorithms used to automatically fix, migrate and manage the systems.  They have a large big data pipeline to process the logs in various ways.

 Privacy Statement

The SQL Server Privacy Statement has been updated. It explains the data collection and use practices for Microsoft SQL Server 2016 (SQL Server) and included or referenced products which can be installed separately (such as SQL Server Management Studio). This is a huge change with Microsoft stating explicitly what data they store.  It explicitly provides a data classification matrix for access control, customer content, end-user identifiable information (EUII) , internet based services data and system metadata and explains permitted usage scenarios, access restrictions and retention requirements. This will massively help with regulatory and compliance requirements for the up and coming GDPR. You can now OPT IN and OPT OUT.

Compliance Boundary Around the Service

The way for security is using more automated ways to protect systems. Microsoft use

• Two factor authentication to cross boundary to administrator / troubleshoot  
•  All access is audited
• Customer data is protected in the boundary
• Telemetry is used to protect data
• Telemetry is scrubbed to protect customer data
•  The disks are encrypted

SQL Server vNext

SQL Server vNext is to be shipped later this year.  We can expect to see the SQL Server products to have more frequent releases than the previous every 2 to 3 years.

Another big change in Microsoft development is for all SQL product features is that they need to have customers asking for them. This sounds like a more relevant model.

The vNext version of SQL Server will have important new features such as being able to

• SQL Server on Linux
• Linux based Docker containers
• SQLGraph
• Cluster-less Availability Groups
• Availability Groups can now work across Windows-Linux to enable cross-OS migrations
• Online non-clustered columnstore index build and rebuild
• Temporal Tables Retention Policy support

 SQL Azure Cloud Service is similar but not quite the same as the SQL Server product.  

Enhancing Business Intelligence with Data Science

The heterogeneous nature of data has resulted in an evolution of the business intelligence platform. The traditional data warehouse architectures are now a part of a greater diverse set of products and tools available for use, to gain insight. This new architecture is in Microsoft Azure, which consists of information management, big data stores, machine learning and analytics and intelligence.

This huge number of tools are known as the Cortana Intelligence Suite. Cortana Intelligence is a platform and a process to perform advanced analytics from start to finish. It is a fully managed business intelligence, big data and advanced analytics offerings. Microsoft have been helping people learn the 14 new tools to explain and show how these fit together by using a mnemonic.

Say it Cortana, Cognitive Services, Bot Framework – intelligent assistant for speech and vision

See it Power BI – interactive report and visualization

Stream it Azure Stream Analytics – real time stream processing

Big it HD Insight – implementation of apache Hadoop

Learn it Azure Machine Learning and MRS – machine learning and R Server engine

Relate it Azure SQL DB, Data Warehouse, DocumentDB -  SQL and NoSQL engines

Store it Azure Data Lake –data storage and distributed processing

Bring it Azure Event Hubs – ingest data for web, IoT and apps

Move it Azure Data Factory – pipeline to move data in and out

Doc it Azure Data Catalog – documentation

Host it Microsoft Azure - IaaS, PaaS or SaaS

These tools are supplemented by a modified process model based on the CRISP-DM (Cross Industry Standard Process for Data Mining). CRISP-DM is a data mining process model that describes commonly used approaches that data mining experts use to tackle problems. CRISP-DM has six major phases. 

The Microsoft team science process is:

There are many tools to get started learning about data science and these a just a few.

A collection of data science tools

https://learnanalytics.microsoft.com/

Code samples

https://github.com/Azure/Cortana-Intelligence-Suite-Industry-Solution-How-To-Guides

https://github.com/Microsoft

Free eBooks from Microsoft Press - Microsoft Virtual Academy

https://mva.microsoft.com/ebooks

• Data Science with Microsoft SQL Server 2016
• Microsoft Azure Essentials: Fundamentals of Azure, Second Edition

Data science track in the Microsoft Professional Program

https://academy.microsoft.com/en-us/professional-program/data-science/

DBaaS - Transforming Database Management

There have been various reports about Database as a Service.

The 451 Research October 2015 report on the state of Database as a Service 

The Report












IBM State of DBaaS 2016 report - Database-as-a-Service

The Report

Data and Database Predictions for 2015

There are many predictions of what data and database advancements will transpire throughout 2015. These are my favourite data and database predictions for 2015.

Top 10 2015 Predictions for Microsoft Azure

PAAS will provide a more economical… the case for IAAS based services is being undermined by easier to use PAAS services.

Why 2015 Will Be Year of Big Data: Oracle's Seven Predictions

Data is a new form of capital. Ultimately, information about people, places and things will truly differentiate enterprises.

IDC Reveals Worldwide Big Data and Analytics Predictions for 2015

Growth in applications incorporating advanced and predictive analytics, including machine learning, will accelerate in 2015. These apps will grow 65% faster than apps without predictive functionality.

Database Predictions for 2015

SQL and NoSQL will become best friends

10 Predictions for Data and Analytics in 2015

Hybrid architectures will become the norm for many organizations,

Gartner Identifies the Top 10 Strategic Technology Trends for 2015

Analytics will take center stage as the volume of data generated by embedded systems increases and vast pools of structured and unstructured data inside and outside the enterprise are analyzed.

Your top predictions for Business Analytics in 2015

LinkedIn Data Culture Discussions: Data storytelling will become the next 'big thing' in Analytics!

DBAinSpace Launch me into Space

This is the video of my bid to be launched into space on the Space Adventures suborbital flight with the competition from Redgate.

VOTE NOW   https://www.dbainspace.com/finalists

Hi, I’m a Senior DBA specialising in SQL Server. I’m very passionate about data and databases. I’m already very excited about the launch of SQL Server 2012 next year, seeing how cloud database services bring agility and elasticity and watching to see how the 2 universes of structured and unstructured data collide.

My name was sent to the moon on the Lunar Reconnaissance Orbitar and is on route to Mars with NASA’s Mars Science Laboratory rover which was launched the other week.

The Redgate DBA in Space competition is an astronomical chance of a lifetime.  Please send me into space.

Database-as-a-Service

Database as a service providers are rapidly increasing. These are a few I found

SQL Azure
SQL Azure delivers cloud database services which enable you to focus on your application, instead of building, administering and maintaining databases. It is built on SQL Server technologies and is a component of the Windows Azure platform.
http://www.microsoft.com/windowsazure/sqlazure/

VMware vFabric Data Director
The first database supported on Data Director is VMware vFabric Postgres 9.0 (vPostgres),..
http://www.vmware.com/products/datacenter-virtualization/vfabric-data-director/overview.html

Project RedDwarf – Database as a Service
http://www.openstack.org/blog/2011/04/announcing-project-reddwarf-database-as-a-service/

Amazon Relational Database Service (Amazon RDS)
Amazon RDS automatically patches and backs up your database, storing the backups for a user-defined retention period and enabling point-in-time recovery.

You benefit from the flexibility of being able to scale the compute resources or storage capacity associated with your relational database instance via a single API call. Amazon RDS supports both MySQL and Oracle Databases
http://aws.amazon.com/rds/

Database 2011 Predictions

Just read an excellent article five-predictions-for-your-database-in-2011 predicting various trends in Databases in 2011. They predict a decline in NoSQL databases and move towards Database as a Service (DBaaS). DBaaS is about an entire change in philosophy producing efficiency, fluidity of resources, scale out, paying for actual usage and being able to be elastic. Elasticity discussed here
http://devcentral.f5.com/weblogs/macvittie/archive/2010/12/01/the-database-tier-is-not-elastic.aspx.

DaaS (Data as a Service) is about data collections in the sky.
DBaaS (Database as a Service) is about providing full database functionality.