Matrix index is out of range for deletion
2 views (last 30 days)
Show older comments
oliver
on 10 Apr 2023
Commented: Walter Roberson
on 10 Apr 2023
my project is sentiment analysis I am trying to follow the tutorial "Create Simple Text Model for Classification"
my database is a list of reviews with labelled sentiment (either 'positive' or 'negative)
I am trying to remove any documents containing no words from the bag-of-words model, and remove the corresponding entries in labels
my code is:
filename = "IMBD_reviews_smol.csv";
data = readtable(filename,'TextType','string');
data.sentiment = categorical(data.sentiment);
cvp = cvpartition(data.sentiment,'Holdout',0.1);
dataTrain = data(cvp.training,:);
dataTest = data(cvp.test,:);
textDataTrain = dataTrain.review;
textDataTest = dataTest.review;
YTrain = dataTrain.sentiment;
YTest = dataTest.sentiment;
documents = preprocessText(textDataTrain);
bag = bagOfWords(documents);
bag = removeInfrequentWords(bag,2);
[bag,idx] = removeEmptyDocuments(bag);
Ytrain(idx) = []; %produces an error
Xtrain = bag.Counts;
mdl = fitcecoc(Xtrain,YTrain,"Learners","linear");
function documents = preprocessText(textData)
documents = tokenizedDocument(textData);
documents = addPartOfSpeechDetails(documents);
documents = removeStopWords(documents);
documents = erasePunctuation(documents);
documents = removeShortWords(documents,2);
documents = removeShortWords(documents,15);
end
7 Comments
Walter Roberson
on 10 Apr 2023
Yes, as I indicated, you are removing all documents from the bag, so your training information becomes empty.
Accepted Answer
Walter Roberson
on 10 Apr 2023
Moved: Walter Roberson
on 10 Apr 2023
filename = "IMBD_reviews_smol.csv";
data = readtable(filename,'TextType','string');
data.sentiment = categorical(data.sentiment);
cvp = cvpartition(data.sentiment,'Holdout',0.1);
dataTrain = data(cvp.training,:);
dataTest = data(cvp.test,:);
textDataTrain = dataTrain.review;
textDataTest = dataTest.review;
Ytrain = dataTrain.sentiment;
Ytest = dataTest.sentiment;
documents = preprocessText(textDataTrain);
bag = bagOfWords(documents);
bag = removeInfrequentWords(bag,2);
[bag,idx] = removeEmptyDocuments(bag);
whos Ytrain idx
Ytrain(idx) = []; %produces an error
Xtrain = bag.Counts;
whos
mdl = fitcecoc(Xtrain, Ytrain, "Learners", "linear");
function documents = preprocessText(textData)
documents = tokenizedDocument(textData);
documents = addPartOfSpeechDetails(documents);
documents = removeStopWords(documents);
documents = erasePunctuation(documents);
documents = removeShortWords(documents,2);
documents = removeShortWords(documents,15);
end
You are removing all of the documents. The bag is left empty.
2 Comments
Walter Roberson
on 10 Apr 2023
You were calling removeShortWords twice, so all words less than 15 characters were being removed. The remaining "words" all happened to be unique, so removing infrequent words resulted in an empty bag.
filename = "IMBD_reviews_smol.csv";
data = readtable(filename,'TextType','string');
data.sentiment = categorical(data.sentiment);
cvp = cvpartition(data.sentiment,'Holdout',0.1);
dataTrain = data(cvp.training,:);
dataTest = data(cvp.test,:);
textDataTrain = dataTrain.review;
textDataTest = dataTest.review;
Ytrain = dataTrain.sentiment;
Ytest = dataTest.sentiment;
documents = preprocessText(textDataTrain);
bag = bagOfWords(documents);
bag = removeInfrequentWords(bag,2);
[bag,idx] = removeEmptyDocuments(bag);
Ytrain(idx) = [];
Xtrain = bag.Counts;
mdl = fitcecoc(Xtrain, Ytrain, "Learners", "linear");
mdl
function documents = preprocessText(textData)
documents = tokenizedDocument(textData);
documents = addPartOfSpeechDetails(documents);
documents = removeStopWords(documents);
documents = erasePunctuation(documents);
documents = removeShortWords(documents,2);
documents = removeLongWords(documents,15);
end
More Answers (0)
See Also
Categories
Find more on Modeling and Prediction in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!