I spoke with support and after some back and forth we were able to trace this problem down to the RowsPerStrip parameter, which is not well explained in the documentation. For anyone else using matlab on BigTIFF files, the default value for this parameter seem to work for input data up to 2GB. Above that the default results in a file that matlab cannot read and you must explicitly set it to a smaller value if you want to eventually open the file in matlab.
I suspect this is actually a memory corruption bug in matlab's Tiff read function (which is why it happens at the 32 bit signed integer boundary), but it is easy to work around so I did not dig into it further.