PASS 24 Data Community Summit

This week it is PASS 24 Data Community Summit

The Microsoft Keynote was entitled, Fuel AI innovation with Azure Databases.

The session abstract: Data is the fuel for innovation, driving the development of transformative AI applications. In this keynote, join Shireesh and the Microsoft engineering team as they explore harnessing the power of data across the Azure databases portfolio to drive ground breaking advancements. Take your first step on the journey with vector search, multi-agent apps and more and learn to unlock new patterns like retrieval augmented generation (RAG).  Explore the latest database innovations in SQL Server, Azure SQL, Azure Cosmos DB, Azure Database for PostgreSQL and Azure Database for MySQL that enhance operational efficiency, deliver personalized user experiences, and revolutionize the way we interact with technology. 

There were some great announcements 

Announcing

• SSMS 21 and Copilot in SSMS
• Azure SQL Database Hyperscale enhancements (Storage up to 128TB (GA) Log throughput rate up to 150 MB per second (Preview) , Continuous priming (Preview))

Preview

• Next gen general purpose on Azure SQL Managed Instance (32TB of storage, 500 DBs, Lower storage latency, improved storage importance, customisable I/O performance)
• Migration Assessment from SQL Server enabled by Azure Arc
• Mirroring in Fabric for Azure SQL Database and Azure Cosmos DB
• In-database embedding generation
• Vector Indexing and search in Azure Cosmos DB for NoSQL
• Native vector type and functions in Azure SQL Database (Available now)
• DiskANN indexing in Azure Database for PostgreSQL

Generally available

• Bi-directional disaster recovery with link feature in Azure SQL Managed Instance
• Flat and quantized flat vector indexes in Azure Cosmos DB for NoSQL
• Serverless auto-pause delay to 15 min
• Azure AI extension
• Hyperscale elastic pools
• Azure Cosmos DB for MongoDB enhancements

PASS Community Summit 2023 Keynote 1

The PASS Data Community Summit 2023 global data conference is happening in Seattle between 14-17 November 14-17 2023.  

The first keynote 'Limitless growth, limitless opportunities: Data and innovation in an AI world' was given by  Shireesh Thota. He started talking about new technologies and how they offer limitless potential with AI, Generative AI and Real-time capabilities.  The limitiless growth with Azure and those SQL databases: SQL Server, Azure SQL, Postgres, CosmosDB and MySQL. These offer limitless scale, flexible tools, Built-in AI, Performance and Guaranteed availability. 

There are celebrations a foot due to SQL Server on Windows reaching 30 years old this year. SQL Server 2022 is the fastest adopted version of SQL Server with growth on both Windows and Linux. He also mentioned SQL Server enabled by Azure Arc with that cloud manageability for managing your entire SQL Server estate. Announcements were made

Discussions moved to Azure SQL and the family of SQL cloud in edge databases. A further announcement was made about the Azure SQL Managed Instance feature wave , that is now generally available. Azure SQL Managed instance has a free offer in preview for 12 months of free managed instance, available soon.

 What's new in Azure SQL Managed Instance?

They talked about that Cloud Modernization Journey: Evaluate, Optimize, Migrate and Modernize with ease. 

Another announcement Azure SQL Database Hyperscale will be same price as commercial OSS databases – General Available mid December.

Some further to announcements were shared

• Azure Cosmos DB for MongoDB vCore – Generally Available
• Vector Search in Azure Cosmos DB for MongoDB vCore – Generally Available 
• Natural language queries for Azure Cosmos DB – Public Preview
• Azure Database for PostgreSQL Extension for Azure AI – Preview

Then concluding with the database of the future will be autonomous and intelligent, developer friendly, cost efficient, and mission critical by default.

PASS Data Community Summit 2023

PASS Data Community Summit have revealed that they will be livestreaming 16 key sessions for free. LIVE from #PASSDataSummit 2023 in Seattle November 14-17.  From #AzureSQL, #SQLServer to #DevOps and #ChatGPT, we've got it all covered.  More details can be found in their blog post  on how to join the livestream. 

All timings are in Pacific time, and the sessions are:

Wednesday November 15

8.15-9.30am – KEYNOTE: Limitless growth, limitless opportunities: Data and innovation in an AI world

9.30-10.15am – break

10.15-11.30am – COMMUNITY: What’s New in Azure SQL and SQL Server, Data Exposed Live

11.30am-12pm – break

12-1pm – LUNCHEON: Database DevOps Luncheon: All you Ever Wanted to Know but were too Afraid to Ask

1-1.30pm – break

1.30-2.45pm – MICROSOFT: Data Warehousing Simplified in Microsoft Fabric

2.45-3.15pm – break

3.15-4.30pm – COMMUNITY: Automating Database Deployments using Azure DevOps

4.30-4.45pm – break

4.45-6pm – AWS: 3-Step Guide to Modernize SQL Server to PostgreSQL

Thursday November 16

8.15-9.30am – KEYNOTE: The New Database Landscape – Revealing Shifts and Charting the Next Horizon

9.30-10.15am – break

10.15-11.30am – REDGATE: Redgate Product Launch: Transform your software delivery with Test Data Management

11.30am-12pm – break

12-1pm – LUNCHEON: Women in Technology: Got your Back: Finding Allies at Work

1-1.30pm – break

1.30-2.45pm – INTEL: How Microsoft & Intel Power Your SQL Server environments

2.45-3.15pm – break

3.15-4.30pm – COMMUNITY: Easy Enterprise Integration with Azure PaaS Services

4.30-4.45pm – break

4.45-6pm – COMMUNITY: Accelerated Database Recovery – A Deep Dive Behind the Magic

Friday November 17

8.15-9.30am – KEYNOTE: AIOps and ChatGPT: Prepare to Ride the Next Wave

9.30-10.15am – break

10.15-11.30am – REDGATE: Cloud Migration: The Good, the Bad, and How to Plan for Success

11.30am-1.30pm – break

1.30-2.45pm – MICROSOFT: What You Should Know About Always On Availability Groups

2.45-3.15pm – break

3:15-4:30pm – MICROSOFT: Everything You Need to Know About Data Virtualization in SQL

Lots of amazing sessions to watch.

Day 2 PASS Data Community Keynote

The keynote is about Doing More with Less: The Challenges Ahead for Every Data Professional. These are challenging times for every business everywhere. So how can you ensure you get the most from your investment in IT, to meet your needs now, while also preparing you for an uncertain future? Join Jakub Lamik, Steve Jones and Kathi Kellenberger in this keynote which looks at how the world is changing for data professionals, and the areas to focus on which will bring the best return on your investment in the long term.

Speakers

Steve Jones | Advocate

Jakub Lamik | CEO

Kathi Kellenberger | Customer Success Engineer

David Bick | Head of Product Marketing

Arneh Eskandari | Solutions Engineering Manager

There were123 sessions yesterday. Make sure you take advantage of the  networking opportunities.

16.10 Steve Jones opens the keynote talking about Kalen the technical people he had met. Opportunity, to learn and be inspired. 

Redgate are expanding across multi databases with there tools. Redgate live the Devops way, build a great culture internally and community centred. SQL Saturday started to bring it to the community. Redgate gave it to PASS. Then donated PASS away after buying it last year. 

The CEO of Redgate is on the stage.  More than a conference: a homecoming. Why important to invest in the community? Database professionals answer important questions about our world.

Every business is a software business - Satya Nadella

Software means data really, how data is ubiquitous and how it enables transformation.  Change is constant.

How has the database landscape changed.

Open source is increasingly important, all sectors and all segments. A powerful tool for community.

Cross database estate is on the rise.  - From the state of the data estate report.

Fragmentation is accelerating. Developing are influencing the databases they use.

PostgreSQL and MySQL are developing the most. SQL Server in the top 6.

Multi database environments are here to stay and complexity is growing - Steve Jones

Micro Services enable agility between software and dev teams - further drives fragmentation

Complexity is mirrored in the environment and automation required with a standard set of tool and approach. This helps with the Devops environment.

David Bick is on the Stage

Just solving SQL Server problems is just not enough for our customers. A lot of postgreSQL  growing. 

Flyway is a database schema for a standard approach using more than 25 databases. From on prem to cloud. Redgate bought and support the open support project.

Gives richer automation. Design principles - should be able to work the way you want to, flexibility, Tools and approach need to be the same but the changes need to be database specific. 

Flyway Enterprise is released for SQL Server and Oracle. This will unlock devops for developers.

Migration to cloud is expanding and Redgate can deploy in hours. Cloud required collaboration. No more silos. DBAs must be database agnostic and support hybrid database estates. 

Arneh  is on the Stage. Security and monitoring is centralised, but individual business units chose different platforms. 

Scalability and being held back . Domain expertise is needed from various cloud technologies. Not enough dashboards across all of these.  Cost control is required to be kept as low as possible. 

A typical financial set up 1000+ database 50/50 split on prem and in the cloud in the UK and US. They need granular monitoring as there is no holistic monitor. 

About solutions, community and support. 

How to help you grow. There are lots of new speakers at Summit this year.  How many children want to be a DBA when they grow up. Simple talk is a great launching point for your career. Louis Davidson is the new editor of simple talk. 

Adoption, results and advocacy in customer success. They are the face of the customer internally and getting value out of there products. Customer success is who Redgate are. Also further ongoing professional development.  There are free industry learning classes which were created in the pandemic.

The debut Redgate 100 most influential in the database community 2022 was shared.

Day 1 Keynote PASS Data Community Summit

The day 1 keynote of the first in person PASS Data Community Summit was packed with exciting data platform innovations across SQL Server and the Microsoft Intelligent Data Platform.

The Keynote: Transform your Data Estate with Microsoft's Intelligent Data Platform.

There are many data challenges today within operational database, analytics and intelligence and data governance. 

 

Data officers are a critical role and it is a challenge to use the data right. A lot of governance tools are not fully integrated. 

Analytics is a core part and must be front and centre not a back end tool. Data Governance is more than creating a catalogue, it is hybrid and multi cloud and the volume of data is increasing. Data Governance needs to be deeply embedded.

The big announcement : SQL Server 2022 is generally available

SQL Server 2022 with embedded data governance. All the data management through Microsoft Purview, lineage tracking is very important. 

There is a new SQL Server pay-as-you-go licensing model 

The annoucements

SQL Server 2022 - GA (Generally Available)

New SQL Server pay-as-you-go licensing model

Link feature for Azure SQL Managed Instance

Backup portability with SQL Server 2022

SQL + Apps Migration Factory offer

Azure Cosmos DB for PostgreSQL - GA

Azure Data Factory SAP Change Data Capture Connector -GA

Azure Synapse Mapping Data Flow for M365 Graph -Public Preview

Azure Synapse Link for SQL

The keynote  then switched to data governance

Improved root cause analysis and traceability with SQL Dynamic lineage is GA

PASS in Seattle

My visit to PASS Data Community summit in Seattle involved my first flight out of the UK since Covid, 3 speaking opportunities and a visit to the Microsoft Reactor in Redmond to speak with the product teams. It is nice to be out travelling to conferences again and networking with the data community. There is so much added value to be had.

 

PASS Data Community Summit Sessions

I have 2 speaking sessions at PASS Data Community Summit on Wednesday 16 November. The live Q&A from my on demand session on: 

The adoption of Data Governance is often marred with thoughts of control and security. Data Governance is about much more in the new world. It is about driving business transformation, knowing what value your data has, and how to remove those data silos, bringing with it a new data culture. In this session, we will start to explore what the new world of Data Governance is and how we help businesses understand the benefits, why it is important, and how to get started in simple agile steps. This session will help you understand why Azure Purview brings exciting features that slot into the world enabling data governance with ease.  I share more about this in a blog post building data governance into everyday processes

A panel session exploring the role of the database in today’s digital transformation and modernization initiatives. What are the experiences of transformation initiatives and why the database has to be included in these innovation drives for the benefit of increased speed, efficiency and risk management. I share more about this in a blog post The two sides of innovation: Data and DevOps

PASS Summit 2022 session schedule

 I am pleased to share my session schedule 

Business Benefits of Good Governance

Summit - Day 1 09:30 AM - 10:00 AM PST

Category: Live  Q&A

Theme: Ahead of the curve

Transformation and Innovation: Why the Database Must be Included

with Steve Jones and Joshua Higginbotham

Summit - Day 1 02:30 PM-03:45 PM PST

Category: Panel

Theme: Ahead of the curve

Data Governance with Azure Purview - Ask the Experts ( Now Microsoft Purview)

With Erwin de Kreuk and Wolfgang Strasser

Summit - Day 3 09:30 AM - 10:45 AM PST

Category: Panel

Theme: Revolutionary

PASS Data Community Summit 2022

I am really excited to announce that I will be speaking at the Summit for the first time on Microsoft Purview. 

I also have privilege to be part of an ask the experts session with Wolfgang, Erwin and Richard. Submit your questions in advance here https://forms.office.com/r/dTP38LnmsJ

Bridge to a new universe: the end-to-end Azure Data Platform

 

Exciting to watch the Day 1 Keynote a Bridge to a new universe: the end-to-end Azure Data Platform delivered by Rohan Kumar and many other people. 

A journey to a new universe is just waiting to inspire innovation, to tap into limitless possibilities and potential. It covered how to shape your data so you can harness its power to find a new galaxy of insights, answers, and predictions.  Some amazing slides and discussion to set you on a new path. 

Three universes bridged together unmatched analytics and insights, limitless cloud data services and unified data governance.

Rohan shared a great quote "If you want to go fast, go alone. If you want to go far, go together. " The SQL Server community has always worked together to achieve some amazing goals. 

Three main data communities were discussed SQLSaturday, Data Saturdays and the Azure Data Community

SQL Server 2022 preview brings with it many new features.  

There are to be two interlinking services Azure Synapse Link and Azure Purview. 

More details were discussed in the keynote but I will share those separately. As ever an inspiring future for us in the data community. 

Read More

Microsoft Ignite book of news

 Announcing SQL Server 2022 preview: Azure-enabled with continued performance and security innovation

PASS Data Community Summit

 

PASS Data Community SUMMIT is November 8-12, 2021 . It is a free online conference and evolved from the original PASS summit. PASS Data Community Summit 2021 is the year’s largest gathering of Microsoft data platform professionals. Join this free online Summit and connect, share, and learn with 1000s of your peers from the global data platform community.

PASS cease operations

I read some sad news. PASS have been an amazing organization for helping me grow my data platform expertise, learn and network in the database world. The summit events will always be something to remember. They have just always been there. It is sad to have another casualty of COVID-19. 

PASS could not have anticipated the impact COVID-19 would have on the world, and on our organization. We are all reeling from what this year has been and done, and it is with heavy hearts that we must share yet one more bit of bad news from the annus horribilis that was 2020. We are saddened to tell you that, due to the impact of COVID-19, PASS is ceasing all regular operations, effective January 15, 2021. With the registration shortfall from PASS Virtual Summit, a lack of cashflow, and considering future obligations that PASS has on the books (i.e. 5 years’ worth of convention center and hotel agreements), PASS has no choice but to cease its operations and pursue dissolution. During our December 3rd, 2020 Board meeting, the PASS Board unanimously approved taking next steps in this unfortunate, but necessary, direction. PASS has engaged experienced insolvency counsel and other professionals. All steps that PASS takes now are under the advice of our independent legal counsel as a part of the insolvency process. This includes communications, debt repayment, asset disbursement, conducting Board business, and all other actions. We encourage you to take full advantage of any access that you have to PASS content between now and January 15, 2021. After that point, PASS servers will cease to function and members will no longer be able to access any PASS resources. So, in the meantime, you can watch sessions on our website, and download session recordings that you’ve purchased or that you have access to as a PASS Pro member. Please take full advantage of this exclusive content while you can. Thank you for your support and engagement over the years. We wish you all the best. The PASS Board

Out of ashes rises something new that shines brightly in the sky. A beacon of hope and new beginnings. Time to create a new free community conference and #sqlfamily home that utilizes technology, not just  Teams and virtual conferencing but create a virtual world with 3d imaginary of yourself for full interaction with ones peers. There needs to be a central hub as it is so important to be a part of the community to share, grow and innovate. That is what makes the #sqlfamily such a strong community.

PASS Virtual Summit 2020 Keynote Day 1

The summit, 10-13 November 2020 is being live streamed through PASSTV.

This year’s first keynote is entitled 'Bringing the future into focus, the end to end Azure Data platform'.

Digital transformation brings change. This change of new technologies can be challenging to learn. For businesses the economies of scale can create efficiencies. The data platform has many elements BI, Analytics and AI, Hybrid Data management, relational databases, Edge and IOT, NoSQL databases, Azure Open Source Database Services (OSS databases) etc. New technologies can help save time. It is good news that the DBA’s key skills are transferable.

Azure services consist of SQL Server on Azure Virtual Machines (for lift and shift and OS level access), Azure SQL Managed Instances (for modernizing existing apps), Azure SQL Database (for build cloud apps). There is also  hyperscale for specific use cases. SQL Server workloads run best on Azure and have patching regimes.

It is good to see that Azure helps customers move at their own pace to the cloud which will enable them to pivot their company with less risk to something new. Azure SQL Server serverless means  pay only for what is required. Azure Edge is an interesting development and is now generally available.

 

With the complex network of applications and systems in the modern data landscape it can be difficult to connect the data to gain value. Data virtualization is an important change.

Azure Arc

Azure Arc is, I think, a real game changer to help with diverse database locations. It is a set of technologies that extends azure management and native data services outside of azure infrastructure to run across your environment even if you can’t migrate to the cloud due to data sovereignty, latency and or regulatory requirements you can still get the efficiency and agility the cloud offers with Azure Arc.

It is a versionless evergreen SQL that ensures you are always current. It provides cloud elasticity on premises which allows optimization of performance of your workloads and dynamically scale up and down without application downtime.   Azure Arc offers unified management which allows you to see your data services running on premises alongside these running on azure through a single pane of glass and manage them using familiar tools like azure portal, azure data studio and azure CLI.

Azure Arc enabled SQL Server Managed Instance and PostgreSQL Hyperscale are in Public Preview

It is possible to use features such as vulnerability assessment and advance threat protection with SQL Defender using the same rules and machine learning algorithms.

The public preview was announced for

•          Azure Cosmos DB – Serverless for all APIs
•          Azure Database for PostgreSQL – Flexible Server
•          Azure Database for MySQL – Flexible Server
•          Azure Cache for Redis – Enterprise

Cloud Scale analytics on Azure with Azure Synapse Analytics and Azure Databricks

 

Announcing public preview of new guided UI for machine learning models. Then to complete the services, Azure Synapse to analyse the data and then report  on it in Power BI completes the stack.

Two Great Conferences

Next week will be a really exciting week with 2 great conferences

Microsoft Ignite 

November 4–8, 2019 | Orlando, FL

Learn innovative ways to build solutions and migrate and manage your infrastructure. Connect with over 25,000 individuals focused on software development, security, architecture, and IT. Explore new hands-on experiences that will help you innovate in areas such as security, cloud, and hybrid infrastructure and development.

PASS Summit
November 5 - 8, 2019 | Seattle, WA

The summit has 3 tracks architecture, data management and analytics. There is interactive training on the latest technologies and spotlights on hot topics such as security, cloud, and AI will be led by the best data minds in technology.

PASS Summit 2018 Day 2 Keynote

The day 2 keynote today was given by Microsoft Data Platform CTO Raghu Ramakrishnan on the internals of our next evolution in engine architecture which will form the foundation for the next 25 years of the Microsoft data platform.

It covered Azure SQL DB Hyperscale. The changing landscape of data has many challenges. How to leverage unbounded storage and elastic compute as well as the perennial problems: size of data operations are slow with long painful recovery times while masking network latencies.

There isn’t one database system that can do it all well. Users need to move data across systems which is slow and complicates governance.

The most challenging for state management is ACID properties, transactional updates, high velocity of data changes and lowest response times. These issue lead to the SQL Hyperscale.

There are various technical themes: full separation of compute and storage, the quorum (log) is complex, uniquely skewed access pattern and network simply extends the memory hierarchy. He shared a newsflash about Multi-Version Timestamp CC rules resulting from 2 phase locking and MVCC (Hekaton) and lock free data structures.

Persistent Version Store (PVS)

This technical in depth talk was packed full of technical content about SQL Hyperscale and I would recommend watching the recording about this new product and era of database delivery.

PASS Summit 2018 Keynote Day 1

The first keynote of PASS summit was delivered by Rohan Kumar entitled SQL Server and Azure Data Services: Harness the ultimate hybrid platform for data and AI. 

Customer priorities for a modernized data estate are: modernizing on-premises, modernizing to cloud, build cloud native apps and unlocking insights.

The announcements follow:

SQL Server 2019

SQL Server 2019 Public Preview  is a great way to celebrate the 25th anniversary of SQL Server. 

There is the introduction of big data clusters which combines Apache Spark and Hadoop into a single data platform called SQL Server. This combines the power of Spark with SQL Server over the relational and non-relation data sitting in SQL Server, HDFS and other systems like Oracle, Teradata, CosmosDB.

There are new capabilities around performance, availability and security for mission critical environments along with capability to leverage hardware innovations like persistent memory and enclaves.

Hadoop, ApacheSpark, Kubernetes and Java are native capabilities in the database engine.

Accelerated data recovery (ADR) was demonstrated and is incredible. It is at public preview.  The benefits of ADR are

• Fast and consistent Database Recovery
• Instantaneous Transaction rollback
• Aggressive Log Truncation

Azure HDInsight 4.0

HDInsight 4.0 is now available in public preview.

There are several Apache Hadoop 3.0 innovations. Hive LLAP (Low Latency Analytical Processing known as Interactive Query in HDInsight) delivers ultra-fast SQL queries. The Performance metrics provide useful insight.

Integration with Power BI direct Query, Apache Zeppelin, and other tools. To learn more HDInsight Interactive Query with Power BI.

Data quality and GDPR compliance enabled by Apache Hive transactions

Improved ACID capabilities handle data quality (update/delete) issues at row level. This means that GDPR compliance requirements can now be meet with the ability to erase the data at row level. Spark can read and write to Hive ACID tables via Hive Warehouse Connector.

Apache Hive LLAP + Druid = single tool for multiple SQL use cases

Druid is a high-performance, column-oriented, distributed data store, which is well suited for user-facing analytic applications and real-time architectures. Druid is optimized for sub-second queries to slice-and-dice, drill down, search, filter, and aggregate event streams. Druid is commonly used to power interactive applications where sub-second performance with thousands of concurrent users are expected.

Hive Spark Integration

Apache Spark gets updatable tables and ACID transactions with Hive Warehouse Connector

There are several Apache Hadoop 3.0 innovations. Hive LLAP (Low Latency Analytical Processing called Interactive Query in HDInsight) for ultra-fast SQL queries. The Performance metrics provide useful insight.

Integration with Power BI Direct Query, Apache Zeppelin, and other tools. To learn more watch HDInsight Interactive Query with Power BI.

Better data quality and GDPR compliance enabled by Apache Hive transactions

Improved ACID capabilities handle data quality (update/delete) issues at row level. GDPR compliance requirements can now be meet with the ability to erase the data at row level. Spark can read and write to Hive ACID tables via Hive Warehouse Connector

Apache Hive LLAP + Druid = single tool for multiple SQL use cases

Druid is a high-performance, column-oriented, distributed data store, which is suited for user-facing analytic applications and real-time architectures. Druid is optimized for sub-second queries to slice-and-dice, drill down, search, filter, and aggregate event streams. Druid is commonly used to power interactive applications where sub-second performance with thousands of concurrent users are expected.

Hive Spark Integration

Apache Spark gets updatable tables and ACID transactions with Hive Warehouse Connector.

Apache HBase and Apache Phoenix

Apache HBase 2.0 and Apache Phoenix 5.0 get new performance and stability features and all of the above have enterprise grade security.

Azure

Azure event hubs for Kafka is generally available

Azure Data Explorer is in public preview.

Azure Databricks Delta is in public preview

• Connect data scientist and engineers
• Prepare and clean data at massive scales
• Build/train models with pre-configured ML

Azure Cosmos DB multi master replication was demoed with a drawing app, Azure Cosmos DB PxDraw

Azure SQL DB Managed Instances will be at General Availability (GA) on Dec 1^st. This provides Availability Groups managed by Microsoft.

Preview of Machine Learning Services with R support in Azure SQL Database

Power BI

Public Preview of Paginated Reports in Power BI Premium Now Available

Power BI delivers dataflows, enterprise reporting, and major updates to Power BI Desktop

The new Dataflows is an enabler for self-service data prep in Power BI

Power BI Desktop November Update

• Follow-up questions for Q&A explorer. It is possible to ask follow-up questions inside the Q&A explorer pop-up, which take into account the previous questions you asked.
• Copy and paste between PBIX files
• New modelling view makes it easier to work with large models.
• Expand and collapse matrix row headers