This event has ended. Visit the official site or create your own event on Sched.
Welcome! We are very excited you will be participating in API World 2016!
View analytic
Monday, September 12 • 12:00pm - 12:50pm
OPEN TALK: Using Spark with Shiny and R Markdown

Sign up or log in to save this to your schedule and see who's attending!

R is well-suited to handle data that can fit in memory but additional tools are needed when the amount of data you want to analyze in R grows beyond the limits of your machine’s RAM. There have been a variety of solutions to this problem over the years that aim to solve this problem in R; one of the latest options is Apache Spark™. Spark is a cluster computing tool that enables analysis of massive, distributed data across dozens or hundreds of servers. 

RStudio recently announced a new open-source package called sparklyr that facilitates a connection between R and Spark using a full-fledged dplyr backend with support for the entirety of Spark's MLlib library. Due to Spark’s ability to interact with distributed data with little latency, it is becoming an attractive tool for interfacing with large datasets in an interactive environment. In addition to handling the storage of data, Spark also incorporates a variety of other tools including stream processing, computing on graphs, and a distributed machine learning framework. Some of these tools are available to R programmers via the sparklyr package. 

In this talk, we’ll discuss how to leverage Spark’s capabilities in a modern R environment. In particular, we’ll discuss how to use Spark within an R Markdown document or even in an interactive Shiny application. We’ll also briefly discuss alternative approaches to working with large data in R and the pros and cons of using Spark.

avatar for Nathan Stephens

Nathan Stephens

Director of Solutions Engineering, RStudio
Nathan has a background in analytic solutions and consulting. He has experience building data science teams, architecting analytic infrastructure, and delivering innovative data products. He is a long time user of R.

Monday September 12, 2016 12:00pm - 12:50pm
DataWeek: Data Stage