How to take Correlation coefficient by Ignoring infinite values from vectors keeping the same size?

I have two vectors X=[1 3 6 inf 4 5 -inf] and Y=[3 5 9 4 inf 23 -inf] with infinite values at random positions in both and want to take corrcoef by keeping the same size but comes out as NaN so how to get a valid value by ignoring infinite values?

[David Young]

I'm not sure what you mean by "keeping the same size". If you ignore the infinite values, the vectors will be length 5 instead of length 7. What answer would you want for the example you give?