This is machine translation

Translated by Microsoft
Mouseover text to see original. Click the button below to return to the English version of the page.

Note: This page has been translated by MathWorks. Click here to see
To view all translated materials including this page, select Country from the country navigator on the bottom of this page.

doc2sequence

Convert documents to sequences for deep learning

Syntax

sequences = doc2sequence(enc,documents)
sequences = doc2sequence(emb,documents)
sequences = doc2sequence(___,Name,Value)

Description

example

sequences = doc2sequence(enc,documents) returns a cell array of the numeric indices of the words in documents given by the word encoding enc. Each element of sequences is a vector of the indices of the words in the corresponding document.

example

sequences = doc2sequence(emb,documents) returns a cell array of the embedding vectors of the words in documents given by the word embedding emb. Each element of sequences is a matrix of the embedding vectors of the words in the corresponding document.

example

sequences = doc2sequence(___,Name,Value) specifies additional options using one or more name-value pair arguments.

Examples

collapse all

Load the weather reports data and create a tokenizedDocument array.

filename = "weatherReports.csv";
data = readtable(filename,'TextType','string');
textData = data.event_narrative;
documents = tokenizedDocument(textData);

Create a word encoding.

enc = wordEncoding(documents);

Convert the documents to sequences of word indices.

sequences = doc2sequence(enc,documents);

View the sizes of the first 10 sequences. Each sequence is a 1-by-S vector, where S is the number of word indices in the sequence. Because the sequences are padded, S is constant.

sequences(1:10)
ans = 10x1 cell array
    {1x930 double}
    {1x930 double}
    {1x930 double}
    {1x930 double}
    {1x930 double}
    {1x930 double}
    {1x930 double}
    {1x930 double}
    {1x930 double}
    {1x930 double}

Convert an array of tokenized documents to sequences of word vectors using a pretrained word embedding.

Load a pretrained word embedding using the fastTextWordEmbedding function. This function requires Text Analytics Toolbox™ Model for fastText English 16 Billion Token Word Embedding support package. If this support package is not installed, then the function provides a download link.

emb = fastTextWordEmbedding;

Load the weather reports data and create a tokenizedDocument array.

filename = "weatherReports.csv";
data = readtable(filename,'TextType','string');
textData = data.event_narrative;
documents = tokenizedDocument(textData);

Convert the documents to sequences of word vectors using doc2sequence. The doc2sequence function, by default, left-pads the sequences to have the same length. When converting large collections of documents using a high-dimensional word embedding, padding can require large amounts of memory. To prevent the function from padding the data, set the 'PaddingDirection' option to 'none'. Alternatively, you can control the amount of padding using the 'Length' option.

sequences = doc2sequence(emb,documents,'PaddingDirection','none');

View the sizes of the first 10 sequences. Each sequence is D-by-S matrix, where D is the embedding dimension, and S is the number of word vectors in the sequence.

sequences(1:10)
ans = 10×1 cell array
    {300×8  single}
    {300×39 single}
    {300×14 single}
    {300×14 single}
    {300×0  single}
    {300×15 single}
    {300×20 single}
    {300×6  single}
    {300×21 single}
    {300×10 single}

Convert a collection of documents to sequences of word vectors using a pretrained word embedding, and pad or truncate the sequences to a specified length.

Load a pretrained word embedding using fastTextWordEmbedding. This function requires Text Analytics Toolbox™ Model for fastText English 16 Billion Token Word Embedding support package. If this support package is not installed, then the function provides a download link.

emb = fastTextWordEmbedding;

Load the weather reports data and create a tokenizedDocument array.

filename = "weatherReports.csv";
data = readtable(filename,'TextType','string');
textData = data.event_narrative;
documents = tokenizedDocument(textData);

Convert the documents to sequences of word vectors. Specify to left-pad or truncate the sequences to have length 100.

sequences = doc2sequence(emb,documents,'Length',100);

View the sizes of the first 10 sequences. Each sequence is D-by-S matrix, where D is the embedding dimension, and S is the number of word vectors in the sequence (the sequence length). Because the sequence length is specified, S is constant.

sequences(1:10)
ans = 10×1 cell array
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}
    {300×100 single}

Input Arguments

collapse all

Input word embedding, specified as a wordEmbedding object.

Input word encoding, specified as a wordEncoding object.

Input documents, specified as a tokenizedDocument array.

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: 'Length','shortest' truncates the sequences to have the same length as the shortest sequence.

Unknown word behavior, specified as the comma-separated pair consisting of 'UnknownWord' and one of the following:

  • 'discard' – If a word is not in the input map, then discard it.

  • 'nan' – If a word is not in the input map, then return a NaN value.

Tip

If you are creating sequences for training a deep learning network with a word embedding, use 'discard'. Do not use sequences with NaN values, because doing so can propagate errors through the network.

Padding direction, specified as the comma-separated pair consisting of 'PaddingDirection' and one of the following:

  • 'left' – Pad sequences on the left.

  • 'right' – Pad sequences on the right.

  • 'none' – Do not pad sequences.

Tip

When converting large collections of data using a high-dimensional word embedding, padding can require large amounts of memory. To prevent the function from adding too much padding, set the 'PaddingDirection' option to 'none' or set 'Length' to a smaller value.

Padding value, specified as the comma-separated pair consisting of 'PaddingValue' and a numeric scalar. Do not pad sequences with NaN, because doing so can propagate errors through the network.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Sequence length, specified as the comma-separated pair consisting of 'Length' and one of the following:

  • 'longest' – Pad sequences to have the same length as the longest sequence.

  • 'shortest' – Truncate sequences to have the same length as the shortest sequence.

  • Positive integer – Pad or truncate sequences to have the specified length. The function truncates the sequences on the right.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

Output Arguments

collapse all

Output sequences, returned as a cell array.

For word embedding input, the ith element of sequences is a matrix of the word vectors corresponding to the ith input document.

For word encoding input, the ith element of sequences is a vector of the word encoding indices corresponding to the ith input document.

Tips

  • When converting large collections of data using a high-dimensional word embedding, padding can require large amounts of memory. To prevent the function from adding too much padding, set the 'PaddingDirection' option to 'none' or set 'Length' to a smaller value.

Introduced in R2018b