partition

Partition minibatchqueue

collapse all in page

    Syntax

    submbq = partition(mbq,numParts,indx)

    Description

    submbq = partition(mbq,numParts,indx) partitions the minibatchqueue object mbq into numParts parts and returns the partition corresponding to the index indx. The properties of submbq are the same as the properties of mbq.

    The output minibatchqueue object has access only to the partition of data it is given when it is created. Using reset with submbq resets the minibatchqueue object to the start of the data partition. Using shuffle with submbq shuffles only the partitioned data. If you want to shuffle the data across multiple partitions, you must shuffle the original minibatchqueue object and then re-partition.

    example

    Examples

    collapse all

    Use the partition function to divide a minibatchqueue object into three parts.

    Create a minibatchqueue object from a datastore.

    ds = digitDatastore;
mbq = minibatchqueue(ds)
    mbq = 
minibatchqueue with 1 output and properties:

   Mini-batch creation:
               MiniBatchSize: 128
            PartialMiniBatch: 'return'
                MiniBatchFcn: 'collate'
    PreprocessingEnvironment: 'serial'

   Outputs:
                  OutputCast: {'single'}
             OutputAsDlarray: 1
             MiniBatchFormat: {''}
           OutputEnvironment: {'auto'}

    Partition the minibatchqueue object into three parts and return the first partition.

    sub1 = partition(mbq,3,1)
    sub1 = 
minibatchqueue with 1 output and properties:

   Mini-batch creation:
               MiniBatchSize: 128
            PartialMiniBatch: 'return'
                MiniBatchFcn: 'collate'
    PreprocessingEnvironment: 'serial'

   Outputs:
                  OutputCast: {'single'}
             OutputAsDlarray: 1
             MiniBatchFormat: {''}
           OutputEnvironment: {'auto'}

    sub1 contains approximately the first third of the data in mbq.

    Use the partition function to divide a minibatchqueue object into three parts.

    Create a minibatchqueue object from a datastore.

    ds = digitDatastore;
mbq = minibatchqueue(ds)
    mbq = 
minibatchqueue with 1 output and properties:

   Mini-batch creation:
               MiniBatchSize: 128
            PartialMiniBatch: 'return'
                MiniBatchFcn: 'collate'
    PreprocessingEnvironment: 'serial'

   Outputs:
                  OutputCast: {'single'}
             OutputAsDlarray: 1
             MiniBatchFormat: {''}
           OutputEnvironment: {'auto'}

    Partition the minibatchqueue object into three parts on three workers in a parallel pool. Iterate over the data on each worker.

    numWorkers = 3;
p = parpool("Processes",numWorkers);
parfor i=1:3
    submbq = partition(mbq,3,i);
    while hasdata(submbq)
        data = next(submbq);
    end
end

    Each worker has access to a subset of the data in the original minibatchqueue object.

    Input Arguments

    collapse all

    Mini-batch queue, specified as a minibatchqueue object.

    Number of partitions, specified as a numeric scalar.

    Partition index, specified as a numeric scalar.

    Output Arguments

    collapse all

    Output minibatchqueue, specified as a minibatchqueue object. submbq contains a subset of the data in mbq. The properties of submbq are the same as the properties of mbq.

    Version History

    Introduced in R2020b

    See Also

    | | |

    Topics