As Walter has mentioned in his answer, inputdig returns data in cells and not numeric values. And therefor cell values can't be directly used with logical operator. You need to convert this cell value to a numerical before applying logical operator.
Apart from that, you are using a while loop to check conditions and if condition true, loop will run. But there is no condition to stop this loop so it will run continuosly. I think in this case, an if loop is sufficient.
BoatStyle = menu('Please select the style the most describes your boat:','Center Console','Viking','Flat Bottom','Yacht','Pontoon');
if BoatStyle == 1
BoatStyle = str2num(cell2mat(inputdlg('Please enter the length of your boat')))
if isempty(BoatStyle) || BoatStyle < 9 || BoatStyle > 61
msgbox('Error. Please input realistic boat dimentions between 10ft - 60ft.')
if BoatStyle >= 10 && BoatStyle <= 60
msgbox('This is a good size boat.')