Spark+AI gems (from the Summit)

Below are videos worth checking out from recent Spark+AI Summit.

Building the Software 2.0 Stack Andrej Karpathy (Tesla) – Andrej’s talk at Spark+AI Summit. If I had time to watch one, I’d do this one.

A lot of our code is in the process of being transitioned from Software 1.0 (code written by humans) to Software 2.0 (code written by an optimization, commonly in the form of neural network training). In the new paradigm, much of the attention of a developer shifts from designing an explicit algorithm to curating large, varied, and clean datasets, which indirectly influence the code. I will provide a number of examples of this ongoing transition, cover the advantages and challenges of the new stack, and outline multiple opportunities for new tooling.

Using AI to Build a Self-Driving Query Optimizer

Scaling Machine Learning at with H2O Sparkling Water and FeatureStore

Time Series Forecasting Using Recurrent Neural Network and Vector Autoregressive Model: When and How

Efficiently Triaging CI Pipelines with Apache Spark :spark:

Dynamic Priorities for Apache Spark Application’s Resource Allocations

Building a Scalable Record Linkage System with Apache Spark, Python 3, and Machine Learning

How Azure Databricks and PySpark Helped Make IOT Analytics a Reality

ML Meets Economics:New Perspectives and Challenges

Project Hydrogen: State-of-the-Art Deep Learning on Apache Spark

(Credits: the list was provided by LSinev, thank you Leonid!)


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s