Data Platform Virtual Summit 2021 Keynote

The Data Platform Summit has started. An excellent session covering the Azure data stack identifying when to use each tool and the key innovations with SQL. It was nice to see an explanation of when to use what tool. With so many tools now it becomes hard to know which is the best choice.  This useful chart was shown.

Azure SQL Edge

SQL Server 2019

Solves the modern data challenges 

• Data visualization and big data clusters
• Modern platforms with compatibility
• Built-in machine learning and extensibility
• Intelligent performance
• Layers of security and complain
• Business critical availability

SQL Linux/Container

Containers are portable and can run anywhere containers are supported. They are lightweight with reduced disk, CPU and memory footprint. They have a consistent image of SQL Server, scripts and tools and are efficient with faster deployment, no patching required and less downtime. 

Azure SQL

SQL Server vs. Azure SQL PasS are 

• business continuity, high availability, automated backups, long term backup retention , geo-replication
• scale, advanced security, version-less, built in monitoring and built-in intelligence. 

There are Azure SQL Editions for general purpose, business critical and hyperscale

Azure Arc

Bring Azure data services to on-premises, multi cloud and edge with Azure Arc. Azure Arc enabled SQL Managed Instance has many advantages.

The value of the Cloud provides additional tools such as

• Azure Defender to protect your data
• Azure SQL Database Ledger for blockchain
• Telemetry across all your assets with Azure Monitor SQL Insights

Then to help with migration there are tools available

• Azure Migrate to discover and assess your SQL Server assets
• Migrate Inline with Azure Data Studio
• Migrate online with Azure Database Migration Service or Log Replay Service

A few tools were not mentioned that form part of the data suite such as Azure Cosmos DB. The expansion of tools and options has grown significantly over the last few years so it is always good to assess what business objective you are trying to achieve and select the right tool.
 

PASS Summit Day 2 SQL Server Evolution

 The day two keynote was delivered by Hanuma Kodavalla a Microsoft Technical Fellow. 

He started with an interesting quote from 'Adventures of a Mathematician' by Stanislaw Ulam

"It is still an unending source of surprise for me to see how a few scribbles on a blackboard or on a sheet of paper could change the course of human affairs."

Then sharing the papers that started it all and the fact that is the 50th anniversary of Codd's paper this year.  

I was interested to learn that the Microsoft Database Research Group is an extension of the SQL product group. 

He mentioned this paper I read a long time ago “One size fits all": an idea whose time has come and gone  M. Stonebraker; U. Cetinteme (2005).  The last 25 years of commercial DBMS development can be summed up in a single phrase: "one size fits all". This phrase refers to the fact that the traditional DBMS architecture (originally designed and optimized for business data processing) has been used to support many data-centric applications with widely varying characteristics and requirements. In this paper, we argue that this concept is no longer applicable to the database market, and that the commercial world will fracture into a collection of independent database engines, some of which may be unified by a common front-end parser.

He went through all the previous versions of SQL with their key features ending with SQL Server 2019.

Another key paper was mentioned

Then moved to discuss newer product features of SQL Azure Serverless and Azure Defender

 

SQL Server secure developments include alway encrypted and secure enclaves.

Then he mentioned the new ledger enabled tables that will be coming soon.

This was a session through history leading to strive forwards to realize Codd's and Gray's vision. It will be exciting to see what comes next. A great session for an industry person who is a database researcher.

Day 1 Microsoft Ignite 2019

Microsoft Ignite day 1 shared many exciting announcements. three of them are

SQL Server 2019 is now generally available.

SQL Server 2019 brings enhancements in the core SQL engine, offers a scale-up and scale-out system with built in support for Big Data (Apache Spark, Data Lake), state of the art data virtualization technology, and with built-in machine learning capabilities.
Get the free e-book on SQL Server 2019.

Simply unmatched, truly limitless:  Azure Synapse Analytics

Azure Synapse Analytics is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives the freedom to query data , using either serverless on-demand or provisioned resources at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs.

Azure services run anywhere with new hybrid capabilities: Azure Arc

Azure Arc is a set of technologies that unlocks new hybrid scenarios for customers by bringing Azure services and management to any infrastructure. Azure Arc is available in preview.

HDFS tiering in SQL Server Big Data Clusters

SQL Server Big data clusters has its own local HDFS built-in data lake to enable the storing of unstructured data and high volume data.  This data virtualization capability has a feature called HDFS tiering. It is a major new contribution to the Apache Hadoop project. 

With HDFS tiering you can access other data lakes by mounting the remote HDFS/S3 compatible data source to your local HDFS data lake. Access is seamlessly available from SQL Server or Apache Spark. Currently you can mount the following storage: Azure Data Lake Storage Gen2, AWS S3, Isilon, StorageGRID and Flashblase.

Introducing SQL Server 2019

SQL Server 2019 is the latest version of  SQL Server. It redefines SQL Server from a traditional relational database system to a data platform for every data scenario from OLTP to DW to now big data and analytics. 

There is a great video on Channel 9 which gives a quick overview of all the new things in SQL Server 2019.

Database Trends Awards

Database Trends and applications have names have listed the best relational database and best big data platform.






Best relational database: SQL Server
"According to Craig S. Mullins, president & principal consultant, Mullins Consulting, Inc, relational continues to dominate: IDC forecasts that relational DBs will still account for more than 80% of the total operational database market through 2022, and Gartner forecasts that through 2020, relational technology will continue to be used for at least 70% of new applications and projects."

Best big data platform: Cloudera Enterprise Data Cloud

"To leverage the immense power of their data, organizations need a solid strategy that incorporates everything from security to data governance to the right big data technologies. Enabling both on-prem and cloud deployments—or a hybrid strategy—big data platforms today support data warehouses, data lakes, data science, engineering, machine learning, myriad database management systems, and much more.  And while Hadoop is a key element of big data platforms today, there are also many other open source components, support capabilities, and advanced features that round out a big data platform to give data-driven companies the big data capabilities they need"

SQL Server 2019 Workshop Lab

SQL Server 2019 is a modern data platform designed to tackle the challenges of today's data professional.

There is a new self-paced free lab is available to learn some of the concepts and how to solve modern data challenges using a hands-on lab approach.

SQL Server 2019 provides many new capabilities including:

• Data Virtualization with Polybase and Big Data Clusters to reduce the need for data movement
• Intelligent Performance to boost query performance with no application changes
• Security enhancements such as Always Encrypted and Data Classification
• Mission Critical Availability including Availability Groups on Kubernetes and Accelerated Database Recovery
• Modern Development capabilities including Machine Learning Services and Extensibility with Java and the language of your choice
• SQL Server on the platform of your choice with compatibility including Windows, Linux, Docker, Kubernetes, and Arm64 (Azure SQL Database Edge)

SQL Bits 2019 Keynote

What an amazing SQLBits in Manchester. Four days packed full of leading edge data technology covering

• SQL Server 2019 Big Data
• Azure SQL Managed Database
• Power BI
• Kubernetes
• Machine Learning
• Python
• Spark

This year SQLBits 2019 had a keynote.  It was nice for the event to have a keynote again. The theme Data Never Rests.  The Microsoft Data Platform Product group who spoke were Buck Woody, Bob Ward, Anna Thomas, Alain Dormehl, Adam Saxton and Patrick LeBlanc. An amazing set of speaks and fun keynote. They shared details of the evolution of the data platform to enable people to keep their skills up to date. The keynote is available to watch . There were several major announcements.

SQL Server 2019 will RTM in second half of the year. SQL Server 2019 CTP2.3 is available now with

• Big data cluster enhancements
• Accelerated database recovery
• Performance enhancements
• Graph data enhancements
• SSAS enhancements

SQL Server 2019 is a modern innovation and there are various forms of the product.

• On Premises
• SQL Server Azure VM (IaaS)
• Azure SQL DB Managed Instance (PaaS)
• Azure SQL Data Warehouse

Azure SQL Database Hyperscale can autoscale up to 100TB and scale compute and storage independently.

During the keynote they showed Azure SQL Database Hyperscale where a 50TB database was restored in just under 8 minutes. That is nice accelerated database recovery.

Data virtualization and big data clusters is a game changing view with SQL Server 2019 big data clusters, data lake scale, machine learning and AI. Multiple data sources can be connected using external table, through the compute pool using Polybase connectors at the source.  Data persistence using multiple data sources is stored in shards of the data pool for SQL Server 2019 big data clusters data mart.

SQL 2019 will send push down predicated queries to other data platforms via Polybase to join SQL data with Oracle, Mongodb and CosmosDB data in one place efficiently.

SQL notebooks in Azure Data Studio is an awesome new feature. 

There is documentation to read and new courses for learning.

aka.ms/DataAccessGuide

and a Summary of All Exams and Certifications Launched in January, 2019!

aka.ms/DataEngCerts

Always Encrypted with Secure Enclaves

SQL Server 2019 preview Always Encrypted uses an enclave technology called Virtualization Based Security (VBS) memory enclaves. A VBS enclave is an isolated region of memory within the address space of a user-mode process.

The capabilities this brings are

• In-place encryption. Encrypt column, rotate a column encryption key, or change an encryption type of a column, without moving your data out of the database
• Rich computations. The engine can delegate some operations on encrypted database columns to the enclave. It can decrypt the sensitive data and execute requested operations in a query on plain text values.

Always Encrypted with secure enclaves allows computations on plain text data inside a secure enclave on the server side. Microsoft define a secure enclave as a protected region of memory within the SQL Server process. It acts as a trusted execution environment for processing sensitive data inside the SQL Server engine. A secure enclave is a black box to SQL Server and other processes on the server. It is not possible to view any data or code inside the enclave from the outside, even with a debugger.

You can now try and evaluate Always Encrypted with secure enclaves in the preview of SQL Server 2019.

This shows what an admin would see when browsing the enclave memory using a debugger (note the question marks, as opposed to the actual memory content).

Reading

Always Encrypted with Secure Enclaves – Try It Now in SQL Server 2019 Preview! 
Always Encrypted with Secure Enclaves

SQL Server 2019, Big Data and AI

At Microsoft Ignite SQL Server 2019 was launched. An amazing product for the future combining SQL Server 2019 with big data and analytics. It is great to see the combining of multiple tools in once place, a one stop shop for large and small data, structured and unstructured and from multiple sources.

There are 3 major components to SQL Server 2019.

The creation of a data virtualization layer that handles complexity of all data sources and format.  Enabling the integration of structured and unstructured data without moving the data.

The streamlining of data management with SQL Server 2019 big data clusters deployed in Kubernetes integrating HDFS and Spark. The architecture is explained in more depth here and looks like

The creation of a complete AI platform that can use Spark to analyse both structured and unstructured data anywhere, use SQL Server machine learning services and SparkML.

In summary SQL Server big data clusters allow you to deploy scalable clusters of SQL Server, Spark, and HDFS Docker containers running on Kubernetes.

Read More

SQL Server 2019 preview combines SQL Server and Apache Spark to create a unified data platform

SQL Server 2019 Early adoption program

What's new in SQL Server 2019?

What is SQL Server 2019 Big Data Clusters?