datastoreToRDD

Class: matlab.compiler.mlspark.SparkContext
Namespace: matlab.compiler.mlspark

Convert MATLAB datastore to a Spark RDD

Syntax

rdd = datastoreToRDD(sc,ds)


Description

rdd = datastoreToRDD(sc,ds) converts a MATLAB^® datastore object ds to a Spark™ RDD.

Input Arguments

sc — SparkContext to use

SparkContext object

The SparkContext to use, specified as a SparkContext object.

ds — Datastore to be converted

MATLAB datastore

Datastore to be converted to an RDD, specified as a MATLAB datastore object.

Output Arguments

rdd — Output RDD

RDD object

Output RDD representing the converted datastore object, returned as a RDD object.

Examples

Convert MATLAB Datastore to Spark RDD

Convert a MATLAB datastore object to a Spark RDD.

% Setup Spark Properties as a MATLAB Map object using a containers.Map class
sparkProp = containers.Map({'spark.executor.cores'}, {'1'}); 

% Create SparkConf object
conf = matlab.compiler.mlspark.SparkConf(...
    'AppName'        , 'myApp', ...
    'Master'         , 'local[1]', ...
    'SparkProperties',  sparkProp );
     
% Create a SparkContext
sc = matlab.compiler.mlspark.SparkContext(conf);

% Create a MATLAB datastore 
ds = datastore('airlinesmall.csv','TreatAsMissing','NA');

% Convert MATLAB datastore to Spark RDD 
rdd = datastoreToRDD(sc,ds);

% Alternate object usage:
rdd = sc.datastoreToRDD(ds);

Version History

Introduced in R2016b