Spark AI Summit 2020

The largest data and machine learning conference, Spark and AI Summit brings together over 7,500 engineers, scientists, developers, analysts and leaders from around the world to San Francisco every year. Over four days, we shape the future of big data, analytics and AI as we share knowledge, hear from thought leaders and train on open-source technologies like Apache Spark, Delta Lake, MLflow, Koalas, TensorFlow and PyTorch. This year the virtual event for data teams has free general admission, 22-26 June.

What is a Lakehouse?

A new management paradigm has emerged that combines data lakes and data warehouses. Lakehouses are similar to data warehouses with structures and data management features. This is backed with the low cost storage that is used for data lakes.

A lakehouse has some key attributes:

• Transaction support
• Schema enforcement and governance
• BI support
• Storage is decoupled from compute
• Openness
• Support for diverse data types ranging from unstructured to structured data
• Support for diverse workloads
• End-to-end streamingThere is a great article to read which covers this in more depth The Data Lakehouse – Dismantling the Hype

DataWorks Summit 2018

This was the first time I had attended the DataWorks summit: Ideas. Insights. Innovation. for big data. I had the privilege to attend the Luminaries dinner on arrival at the conference. The dinner was held for the European data heroes award. The Hortonworks data heroes initiative recognizes the data visionaries, data scientists, and data architects transforming their businesses and organizations through Big Data.

Each day started with a set of keynotes.

Day 1 Opening Keynotes

The Single Most Important Formula for Business Success Scott Gnau - Hortonworks

Changing the Data Game with Open Metadata and Governance Mandy Chessell - IBM

Big Data Success In Practice: The Biggest Mistakes To Avoid Across The Top 5 Business Use Cases Bernard Marr - Bernard Marr & Co.

Munich Re: Driving a Big Data Transformation Andreas Kohlmaier - Munich Re

Scott Gnau opened his talk with an hypothesis “Data is your cloud is your business” Connecting disparate data to provide for real time information enables us to innovated fast. A data strategy is imperative, it needs to include governance, security and adopt rapid change. Data drives our lives everyday from smart edge devices to all businesses.

He concluded with your data strategy is your cloud strategy is your business strategy if (A) =(B) and (B) = (C) then (A) =(C).

Bernard Marr then shared his insights about AI automating more things faster and the fourth industrial revolution.  He mentioned the top 5 business use cases as

• Informing: to make better decisions
• Understand: know you customers better
• Improvement: customer value proposition
• Automation: key business processes
• Monetization: data as an asset

A couple of interesting points raised were about specialist data hunting units to find new data sources and automation requirements to improve operations.  Data diversity is key to improve analytics along with data governance.

Day 2 Keynotes

Renault: A Data Lake Journey Kamelia Benchekroun - Renault Group

Are You Ready For GDPR? Jamie Engesser - Hortonworks, Srikanth Venkat - Hortonworks Inc

Embracing GDPR to Improve Your Business Practices in the Digital Age Enza Iannopollo - Forrester Research

Driving High Impact Business Outcomes from Artificial Intelligence Frank Saeuberlich – Teradata

Day 2 Forester Enza Iannopollo discussed embracing GDPR to improve your business practices in the digital age. Privacy by design and by default requires new business processes to be established and cultural change to happen. GDPR requires compliance across the organization and with external partners. The compliance strategies are only as good as your risk assessment and mitigation. The classification of data is a key place to start. Concluding the sessions with a quote

“Good Data protection normally enables you to do more things with data, not less” Tim Gough Head of Data Protection Guardian News and Media

Data Steward Studio (DSS) was launched at the conference. It is one of several services available for Hortonworks DataPlane Service; it provides a suite of capabilities that allows users to understand and govern data across enterprise data lakes.

Microsoft DataAmp – SQL Server 2017

The DataAmp webcast was packed full of announcements. The Webcast was delivered by Scott Guthrie and Joseph Sirosh. The SQL Server product delivering intelligence, trust and flexibility.

Microsoft confirmed that the next version of SQL Server is SQL Server 2017 and will be available simultaneously on Windows, Linux and Docker. Download the SQL Server 2017 datasheet.  It will be the first RDBMS to deliver AI with data. There is a convergence of cloud, data and intelligence. Delivering AI with data: the next generation of Microsoft’s data platform blog shares more information.  SQL Server 2017 Community Technology Preview 2.0 now available.

There were so many new features announced only a few are mentioned below.  There are adaptive query processing improvements which will enhance the performance of workloads. There is a You Tube video SQL Server 2017: Adaptive Query Processing discussing this. Threat detection is now in Azure SQL Database and is straight forward to configure. 

Hybrid Cloud just got easier to adopt with the new Azure migration resources and tools. To help with SQL Server migrations to the cloud features such as Service Broker, SQLAgent, Profiler etc. are now available in Azure. There is a new data migration service for automatic migration for SQL Server, Oracle and MySQL in Azure .

SQL Graph

Storing and analyzing graph data relationships. This includes full CRUD support to create nodes and edges and T-SQL query language extensions to provide multi-hop navigation using join-free pattern matching.  The SQL Server engine integration enables querying across SQL tables and graph data.

SQL Server on Linux

 

The official Microsoft repository for SQL Server in Docker containers is here.

Here are a few videos to help get you started with Linux:

• SQL Server 2017: Security on Linux
• Get started developing apps with SQL Server 2017 running on Linux
• SQL Server 2017: HA and DR on Linux

Analytics 

SQL Server is the first commercial database to include Deep Learning algorithms, with the announcement of the Microsoft Cognitive Services general availability of the FACE API and Computer Vision API.

Azure Data Lake Services now have petabyte scale.

Azure Analysis Services became generally available. You Tube video: SQL Server 2017: BI enhancements.  

You can use Python for advanced analytics, You Tube video: SQL Server 2017: Advanced Analytics with Python

Azure DocumentDB

Azure DocumentDB is globally distrubuted and offer limitless scale of throughput and storage. It can be used for things such as IoT applications that need low response times and need to handle massive amounts of reads and writes.

Cortana Intelligence Solution Templates

You can now quickly build Cortana Intelligence Solutions from preconfigured solutions, reference architectures and design patterns. Some are released with more to follow.

The really important thing that I am excited about is the flexibility of choice within the SQL Server product.

Joseph Sirosh concluded comparing the industrial revolution with the intelligence revolution of today.