Find decision tree with highest validation accuracy in tree ensemble

Hi,

I'm curious if there is a way to find the most "representative" tree within a ensemble of trees (random forest)? Such tree could be the one which has the highest valdation accuracy within the ensemble when validated on a test set...

Thank you!

Kai

Answer 1

[Abhipsa]

Hi @Kai Doenges,

You can find the single decision tree with the highest validation accuracy by evaluating each tree in the ensemble individually on a validation set.

The below code snippet demonstrates the same:

% You can load your dataset
% I am using Iris dataset which is one of the MATLAB's built in dataset
load fisheriris;
X = meas;  % Features
Y = species;  % Target labels 

% Then, split the dataset into training and validation sets (70% training, 30% validation)
cv = cvpartition(Y, 'HoldOut', 0.3);
XTrain = X(training(cv), :);
YTrain = Y(training(cv));
XValidation = X(test(cv), :);
YValidation = Y(test(cv));

% train an ensemble model using 'fitcensemble' with bagging method
ens = fitcensemble(XTrain, YTrain, 'Method', 'Bag');

% loop through each tree in the ensemble to find the one with the highest validation accuracy
numTrees = ens.NumTrained;
bestAccuracy = 0;
bestTreeIdx = 0;

for i = 1:numTrees
    % Get the i-th decision tree from the trained ensemble
    tree = ens.Trained{i};
    
    % Predict on the validation data using the current tree
    predictedLabels = predict(tree, XValidation);
    
    % Calculate accuracy on the validation set
    acc = mean(strcmp(predictedLabels, YValidation));  % Compare strings for accuracy
    
    % Update if this tree has better accuracy
    if acc > bestAccuracy
        bestAccuracy = acc;
        bestTreeIdx = i;
    end
end

% Display the best decision tree
bestTree = ens.Trained{bestTreeIdx};
disp(['Best Tree Index: ', num2str(bestTreeIdx)]);
disp(['Best Tree Validation Accuracy: ', num2str(bestAccuracy)]);

Moreover, you can utilize the “view” function in MATLAB to visualize the decision tree.

% Visualize the best decision tree
view(bestTree, 'Mode', 'graph');

The below figure represents the best decision tree obtained from the above code snippet.

You can use the below MATLAB commands to learn more about the functions:

>>doc cvpartition
>>doc fitcensemble
>>doc predict
>>doc view

Hopefully this answers your query. Happy Coding!