Apache Spark Under the Hood Free ebook

Databricks has created an ebook to share excerpts from the book, Spark: The Definitive Guide. 

In this eBook, it cover:

• The past, present, and future of Apache Spark.
• Basic steps to install and run Spark yourself.
• A summary of Spark's core architecture and concepts.
• Spark's powerful language APIs and how you can use them.

Azure Open Datasets

Azure Open Datasets are curated public datasets that can be used to add scenario-specific features to machine learning solutions for more accurate models. Open Datasets are on Microsoft Azure and are available to Azure Databricks, Machine Learning service, and Machine Learning Studio. Access to the datasets is through the APIs and other products, such as Power BI and Azure Data Factory.

Spark+AI Summit 2019

The SparkAI Summit shared a lot of  announcements. The open source announcements were






Koalas - a more complete Pandas API

The open sourcing of Databricks Delta as Delta Lake. Delta dramatically simplifies building reliable data lakes on HDFS and cloud storage with ACID transactions, indexes and scalable metadata handling.

Microsoft is joining the MLflow project and adding MLflow APIs in Azure ML.

Rohan Kumar  of Microsoft announced .NET for Apache Spark, making Apache Spark accessible to .NET developers - Git Hub

Spark 3.0 expected later in the year

The keynote videos are all online now and other session videos will be there in about 2 weeks.

SQL Bits 2019 Keynote

What an amazing SQLBits in Manchester. Four days packed full of leading edge data technology covering

• SQL Server 2019 Big Data
• Azure SQL Managed Database
• Power BI
• Kubernetes
• Machine Learning
• Python
• Spark

This year SQLBits 2019 had a keynote.  It was nice for the event to have a keynote again. The theme Data Never Rests.  The Microsoft Data Platform Product group who spoke were Buck Woody, Bob Ward, Anna Thomas, Alain Dormehl, Adam Saxton and Patrick LeBlanc. An amazing set of speaks and fun keynote. They shared details of the evolution of the data platform to enable people to keep their skills up to date. The keynote is available to watch . There were several major announcements.

SQL Server 2019 will RTM in second half of the year. SQL Server 2019 CTP2.3 is available now with

• Big data cluster enhancements
• Accelerated database recovery
• Performance enhancements
• Graph data enhancements
• SSAS enhancements

SQL Server 2019 is a modern innovation and there are various forms of the product.

• On Premises
• SQL Server Azure VM (IaaS)
• Azure SQL DB Managed Instance (PaaS)
• Azure SQL Data Warehouse

Azure SQL Database Hyperscale can autoscale up to 100TB and scale compute and storage independently.

During the keynote they showed Azure SQL Database Hyperscale where a 50TB database was restored in just under 8 minutes. That is nice accelerated database recovery.

Data virtualization and big data clusters is a game changing view with SQL Server 2019 big data clusters, data lake scale, machine learning and AI. Multiple data sources can be connected using external table, through the compute pool using Polybase connectors at the source.  Data persistence using multiple data sources is stored in shards of the data pool for SQL Server 2019 big data clusters data mart.

SQL 2019 will send push down predicated queries to other data platforms via Polybase to join SQL data with Oracle, Mongodb and CosmosDB data in one place efficiently.

SQL notebooks in Azure Data Studio is an awesome new feature. 

There is documentation to read and new courses for learning.

aka.ms/DataAccessGuide

and a Summary of All Exams and Certifications Launched in January, 2019!

aka.ms/DataEngCerts

PASS Summit 2018 Keynote Day 1

The first keynote of PASS summit was delivered by Rohan Kumar entitled SQL Server and Azure Data Services: Harness the ultimate hybrid platform for data and AI. 

Customer priorities for a modernized data estate are: modernizing on-premises, modernizing to cloud, build cloud native apps and unlocking insights.

The announcements follow:

SQL Server 2019

SQL Server 2019 Public Preview  is a great way to celebrate the 25th anniversary of SQL Server. 

There is the introduction of big data clusters which combines Apache Spark and Hadoop into a single data platform called SQL Server. This combines the power of Spark with SQL Server over the relational and non-relation data sitting in SQL Server, HDFS and other systems like Oracle, Teradata, CosmosDB.

There are new capabilities around performance, availability and security for mission critical environments along with capability to leverage hardware innovations like persistent memory and enclaves.

Hadoop, ApacheSpark, Kubernetes and Java are native capabilities in the database engine.

Accelerated data recovery (ADR) was demonstrated and is incredible. It is at public preview.  The benefits of ADR are

• Fast and consistent Database Recovery
• Instantaneous Transaction rollback
• Aggressive Log Truncation

Azure HDInsight 4.0

HDInsight 4.0 is now available in public preview.

There are several Apache Hadoop 3.0 innovations. Hive LLAP (Low Latency Analytical Processing known as Interactive Query in HDInsight) delivers ultra-fast SQL queries. The Performance metrics provide useful insight.

Integration with Power BI direct Query, Apache Zeppelin, and other tools. To learn more HDInsight Interactive Query with Power BI.

Data quality and GDPR compliance enabled by Apache Hive transactions

Improved ACID capabilities handle data quality (update/delete) issues at row level. This means that GDPR compliance requirements can now be meet with the ability to erase the data at row level. Spark can read and write to Hive ACID tables via Hive Warehouse Connector.

Apache Hive LLAP + Druid = single tool for multiple SQL use cases

Druid is a high-performance, column-oriented, distributed data store, which is well suited for user-facing analytic applications and real-time architectures. Druid is optimized for sub-second queries to slice-and-dice, drill down, search, filter, and aggregate event streams. Druid is commonly used to power interactive applications where sub-second performance with thousands of concurrent users are expected.

Hive Spark Integration

Apache Spark gets updatable tables and ACID transactions with Hive Warehouse Connector

There are several Apache Hadoop 3.0 innovations. Hive LLAP (Low Latency Analytical Processing called Interactive Query in HDInsight) for ultra-fast SQL queries. The Performance metrics provide useful insight.

Integration with Power BI Direct Query, Apache Zeppelin, and other tools. To learn more watch HDInsight Interactive Query with Power BI.

Better data quality and GDPR compliance enabled by Apache Hive transactions

Improved ACID capabilities handle data quality (update/delete) issues at row level. GDPR compliance requirements can now be meet with the ability to erase the data at row level. Spark can read and write to Hive ACID tables via Hive Warehouse Connector

Apache Hive LLAP + Druid = single tool for multiple SQL use cases

Druid is a high-performance, column-oriented, distributed data store, which is suited for user-facing analytic applications and real-time architectures. Druid is optimized for sub-second queries to slice-and-dice, drill down, search, filter, and aggregate event streams. Druid is commonly used to power interactive applications where sub-second performance with thousands of concurrent users are expected.

Hive Spark Integration

Apache Spark gets updatable tables and ACID transactions with Hive Warehouse Connector.

Apache HBase and Apache Phoenix

Apache HBase 2.0 and Apache Phoenix 5.0 get new performance and stability features and all of the above have enterprise grade security.

Azure

Azure event hubs for Kafka is generally available

Azure Data Explorer is in public preview.

Azure Databricks Delta is in public preview

• Connect data scientist and engineers
• Prepare and clean data at massive scales
• Build/train models with pre-configured ML

Azure Cosmos DB multi master replication was demoed with a drawing app, Azure Cosmos DB PxDraw

Azure SQL DB Managed Instances will be at General Availability (GA) on Dec 1^st. This provides Availability Groups managed by Microsoft.

Preview of Machine Learning Services with R support in Azure SQL Database

Power BI

Public Preview of Paginated Reports in Power BI Premium Now Available

Power BI delivers dataflows, enterprise reporting, and major updates to Power BI Desktop

The new Dataflows is an enabler for self-service data prep in Power BI

Power BI Desktop November Update

• Follow-up questions for Q&A explorer. It is possible to ask follow-up questions inside the Q&A explorer pop-up, which take into account the previous questions you asked.
• Copy and paste between PBIX files
• New modelling view makes it easier to work with large models.
• Expand and collapse matrix row headers

Machine Learning on Azure

At Microsoft Ignite there were many data announcements. Azure AI is another such area that covers the next wave of innovation aimed at transforming business. There are 3 solution areas. 

Predictive models to optimise business process

These are a set of pretrained models for Azure Cognitive Services and ONNX (Open Neural Network Exchange) that enables model interoperability across frameworks. Machine Learning is available with Azure Databricks, Azure Machine Learning and Machine Learning VMs

AI powered apps to integrate vision, speech and language
There are now services specifically designed to help build AI powered apps & agents.

Knowledge mining to uncover insight from documents
There is valuable information hidden in documents, forms, pdfs and images. Azure Cognitive Search (in preview) adds Cognitive Services on top of Azure Search. 

  

Reading
Azure AI – Making AI real for business

Azure Data Factory Data Flows

There is a preview for a new feature called 'Data Flows' that enables visual data transformation capability within Azure Data Factory (ADF). This is scaled out using Databricks. This is available in the ADF Resource Explorer. Data Flow should enable a deeper understanding of your data. Monitoring and inspecting pipelines can be done in ADF. The transformations occur inside Azure Databricks which uses the Spark engine to distribute data across partitions. 

References

Microsoft Azure Data Factory documentation for Data Flow during preview is available here: https://aka.ms/adfdataflowdocs

Azure Data Factory: Visual Data Flows for Data Transformation (Preview)

There are revised patterns available for big data advanced analytical capabilities using the Azure Databricks platforms with Azure Machine Learning.

The new capabilities will enable advance analytics to be carried out using Azure Machine Learning. The different types of data requirements and consumption are integrated using CosmosDB.