Why does my prediction accuracy vary with different data division technique?

I am currently training various classification algorithms and neural networks with the classification and neural network toolbox. When importing my dataset to the workspace I choose different data division methods.

For my first experiment I use the dividerand method to divide my data into 80% training and 20% test data. After testing a fine KNN for example I get about 98% accuracy for training and about the same for later prediction on the test data.

For my second experiment I use the divideind method to manually choose the train and test volumes. I pick around the same proportion (80%|20%). In this case I get about 98% for the training and just 65% prediction accuracy on the test set. My first thought was: ok, maybe, for some reason the test data only contains the part of way different values regarding the whole data set.

So I do another experiment: In this case I import my data to the workspace, shuffle the data manually with

random_Data = Data(randperm(size(Data, 1)), :);

and only after that pick my ~80% training and ~20% test set with divideind. The result is comparable with the first experiment: ~98% accuracy for training and test.

To be honest, I don't know which result represents the true result. I have ~1.5 million samples in my dataset, which means that the probability of a homogeneous distribution of the data for training and test is relatively small, for which reason i expected the same results for the first and second experiment. If the second result would represent the true result, i would have an extremly overfitted knn model. Further this would mean, at least for my undestanding, that there is something going wrong the way matlab handles randomly divided data. Does anybody have an answer for my problem?

Thanks for any help,

Adrian

PS: For new test data I get for all cases bad classifications around 65%, which kind of supports my theory of overfitting and strange behaviour of matlab with random division.

[Greg Heath]

 1. How much data??   N = ...
 2. Dimensionality?   d = ...
 3. How many classes? c = ...
 4. No. of vectors in each class?
 5. Equalize class sizes using within-class interpolation 
 6. Repeat as many times as necessary
   a. Randomly divide each class into 10 subsets 
   b. Design classifiers using 9 subsets (8 trn + 1 val) 
      from each class
   c. Obtain the test performance on remaining subclasses

Hope this helps.

Greg

[Adrian]

Hi Greg,

I have ~1.5 Million samples, 8 Input Parameters, which are scalars, and 3 classes, which are scalars, too.

I don't understand the further steps and how they are related to my question, can you explain a little more please?

I tried various configurations of the data division with different distributions, but none of them let to another result.

[Viren Gupta]

What is the class distribution ratio on the original dataset and in the training and test dataset in your second experiment?