I figured out one way:
colors = NaN(max(LL(:)),4);
for i = 1:max(LL(:)) % LL is my labelmatrix
[r,c] = find(LL==i,1)
colors(i,1) = i;
colors(i,2) = RGB(r,c,1) % RGB contains the colors generated by 'turbo' and 'shuffle'
colors(i,3) = RGB(r,c,2)
colors(i,4) = RGB(r,c,3)
end