State of the SQL Nation and the Microsoft Engineering Model

SQLBits 2017 was different this year and instead of a keynote launching the event they had a 15 mins Q & A session with Conor Cunningham and Simon Sabin. They discussed the rapid change of new engineering features in the product. There are now new features shipped monthly. The new model requires running test 24/7 and the interesting change is that all builds have the same level of sign off, increasing the quality of the build now. This is why the quality of the 2 monthly cumulative updates has increased. They run over 700,000 functional automated tests and stress test as well. They aim to ship with the minimum viable product and release updates every 3 to 6 months on them. They don’t deprecate features specifically unless there is a security risk and they use telemetry to help measure feature adoption.

SQL Server / Azure Engineering Model

A second session I attended was about the completely new engineering model that Microsoft now operate. There were some very interesting changes that I think all businesses need to think about when designing and structuring new teams to adapt to the changing model.

Microsoft DevOps Interesting Facts

• Microsoft have around 1.7 million production databases.
• They have no Database Administrators (DBAs).
• They have no Operations team.
• Engineers automate everything.
• SQL Server Azure runs 100% DevOps.
• SQL Server is managed by telemetry. 600TB of logs are collected every day and machine learning algorithms used to automatically fix, migrate and manage the systems.  They have a large big data pipeline to process the logs in various ways.

 Privacy Statement

The SQL Server Privacy Statement has been updated. It explains the data collection and use practices for Microsoft SQL Server 2016 (SQL Server) and included or referenced products which can be installed separately (such as SQL Server Management Studio). This is a huge change with Microsoft stating explicitly what data they store.  It explicitly provides a data classification matrix for access control, customer content, end-user identifiable information (EUII) , internet based services data and system metadata and explains permitted usage scenarios, access restrictions and retention requirements. This will massively help with regulatory and compliance requirements for the up and coming GDPR. You can now OPT IN and OPT OUT.

Compliance Boundary Around the Service

The way for security is using more automated ways to protect systems. Microsoft use

• Two factor authentication to cross boundary to administrator / troubleshoot  
•  All access is audited
• Customer data is protected in the boundary
• Telemetry is used to protect data
• Telemetry is scrubbed to protect customer data
•  The disks are encrypted

SQL Server vNext

SQL Server vNext is to be shipped later this year.  We can expect to see the SQL Server products to have more frequent releases than the previous every 2 to 3 years.

Another big change in Microsoft development is for all SQL product features is that they need to have customers asking for them. This sounds like a more relevant model.

The vNext version of SQL Server will have important new features such as being able to

• SQL Server on Linux
• Linux based Docker containers
• SQLGraph
• Cluster-less Availability Groups
• Availability Groups can now work across Windows-Linux to enable cross-OS migrations
• Online non-clustered columnstore index build and rebuild
• Temporal Tables Retention Policy support

 SQL Azure Cloud Service is similar but not quite the same as the SQL Server product.