How can i change a Lat, Long and ellipsoid separation ascii file into a matrix?

Hi all, sorry if this is a basic question.

I currently have a large data set containing Latitudes, longitudes and separation height. the current format of the text file is space delimited in columns as follows

Column A Latitudes, Column B Longitudes, Column C Ellipsoid separations,

I have created a script to evaluate the ellipsoidal separation or any position within the above model, but i cant get the raw data stored in the above text files in the correct format, instead of them being in columns i need column A to become Row one with all latitudes along the top row Column B to become Column A and the relevant separation heights for each of the corresponding latitudes and longitudes to fill the rest of the matrix.

Basically the matrix should read like a football table where you match a latitude from the top row and a longitude from the first column join the two up and that gives you the ellipsoid separation at that location.

so if anyone could help me with a simple script or syntax that would create the matrix im looking for it would be much appreciated.

Thanks Gavin

[dpb]

Is column "C" actually length(A)*length(B)????

If not, you don't have sufficient data to do the requested.

ERRATUM:

OK, I reread the posting and realize I misread "elevation" as "separation" in thinking. See updated response in Answer...

Answer 1

[dpb]

S=zeros(size(D,1)+1);        % initialize array
S(1,2:end)=D(:,1)';          % first column D--> row 1
S(2:end,1)=D(:,2);           % second column-->first

Now call your routine to fill in the array of computed dimensions w/ origin at (2,2). If you've vectorized it, you should be able to write

S(2:end,2:end)=computeS(D);

NB: that to do what you asked you have an array that's one column and row larger than the sizes of the data vectors and the (1,1) location in the result is empty of any useful data. In this case I left it as a zero; you can put in a NaN or whatever.

You can't have it empty and use a regular array; to do that you'd have to go to a cell array where the first row and column might be held as elements and the values as another but using that organization would likely be less convenient altho if you intend to print this out in table form you'll have to write the first row independently of the remainder if you don't want the odd value displayed.

[Gavin]

Hi dpb,

Thanks for the response the first part of the script you have given me seem to work fine S=zeros(fliplr(size(D))+1); % initialize array and S(1,2:end)=D(:,1)'; % first column D--> row 1 put the Latitudes along the first row but the final part S(2:end,1)=D(:,2)'; % second column-->first is returning me the error message "Subscripted assignment dimensions mismatch"

Any ideas??

ps. just to check im not being stupid im still rather new to matlab, D is representing my raw data set in the column vector

Thanks Gavin

[dpb]

Yeah, for some reason it had dawned on me there was an error -- the new matrix needs must be square of [length(D)+1] in each direction since you've only three columns. See the edited/corrected Answer above.

Answer 2

[Gavin]

HI dpb,

The script works now so thank you very much the lats transpose along the top row, the longs down the first column but im still having trouble making the third column of data (separation heights) fill in the rest? these data points for separation values are supposed to fill the middle of the matrix and show the separation at each location?

then i can apply the model in the correct format to interpolate my positions within the master data set and return the correct separation value at the given location?

Thanks for all your help so far and any further assistance you can offer would be greatly appreciated.

Regards Gvin

Answer 3

[dpb]

That's back to the problem as I first interpreted it -- if you have three columns of the same length, there's no way to fully populate an NxN array w/ only N values.

If it is that the third column actually is the result rather than an input to compute a result as I earlier presumed then it will be the diagonal from 2,2 to end,end of the augmented array we called S earlier. In that case then it's simpler to create it in a slightly different fashion...

S=diag([0 D(:,3)']);         % 3rd column on diagonal of (N+1)-square array
S(1,2:end)=D(:,1)';          % first column D--> row 1
S(2:end,1)=D(:,2);           % second column-->first

[Gavin]

Thanks dpb,

The newest version the the script you supplied has worked flawlessly, the text file is now laid out in the format that i require, allowing for surface plots of the ellipsoid seraration from various models. There are still a few unforeseen kinks that i need to over come before my model works. Such as now the text file is in the correct format im trying to use

lat=S(1,1:end-1); lon=S(2:end,1); sep=S(2:end,2:end); to define out the variable and then entering the positions i require separation values for as e and n and running the script

results=interp2(lat,lon,sep,e,n,'cubic');

But i keep getting the error that "the vectors must be monotonic or the matrix created in meshgrid", believe that as my data set is in latitudes and longitudes (with some negative values) and neither variable sorted into ascending order that the above script wont work. I may try converting the lats and longs to UTM and see if the set works better that way.

Again thank you for all your help

Gavin

[dpb]

Indeed, if the independent values aren't monotonic none of the builtin interpX functions will function (so to speak :) ).

The sign won't matter if you can rearrange them to be from minimum to maximum (or converse). It's possible that can't be done simultaneously w/ the data you have.

I've a couple of questions, however, on the idea above, anyway.

1.

lat=S(1,1:end-1);  % Shouldn't be 1,2:end as S(1,1)==>0?

2.

sep=S(2:end,2:end); % Isn't everything but diag(S)==>0?

Where/how did you get anything else to use other than D(:,3) for the nondiagonal values? If so, why not just use the D vectors directly?

I'm still not following how you intend this interpolation scheme to work with only the one set of intersecting data if that is, indeed, all there is. Otherwise, you have some other way to compute the off-diagonal points as I thought must have been the case initially and then you're interpolating between those? Me cornfoozed... :)

Converting to UTM won't solve the problem of nonmonotonic inputs. Can you reorder to achieve that?

doc sort  % NB: the 'rows' optional input may help

Answer 4

[dpb]

From an offline discussion wherein Gavin supplied the actual form of the input file which is a grouping of readings instead of a single series as assumed above, I thought I'd write out the final resolution to build the 2D array as wanted...

 U1=unique(D(:,1));             % The first column unigue values
 U2=unique(D(:,2));             % Ditto, second

Note that unique has the side effect (desired here for later use by interp2) of sorting the output order in increasing magnitude.

 N1=length(U1);  N2=length(U2); % and the number of each
 S=zeros(N1+1,N2+1);            % big enough for the headings, too
 S(1,2:end)=U2;                 % header row
 S(2:end,1)=U1;                 % and column

Now all that is left is to correlate the locations in the original table with their location in the new. An easy way to generate the lookup table is...

 [~,ix1]=histc(D(:,1),U1);          % the bin for each unique value in 
 [~,ix2]=histc(D(:,2),U2);          % new array

Now all that is left is to turn those into locations in the new array...

 ix=sub2ind(size(S),ix1+1, ix2+1);  % bin subscripts to linear indices
 S(ix)=D(:,3);                      % and store...

NB the "plus one" to account for the amended row/column in S compared to the original array size.

ERRATUM:

Corrected the previously-reversed row/column indices into sub2ind

[Gavin]

Hi dpb,

Sorry for late response the new version of the script seems to work very well except the final step? ix=sub2ind(size(S),ix2+1, ix1+1);

I'm getting the error

Error using sub2ind (line 56) Out of range subscript.

This has begun to move beyond my understanding.. thank you for all your help so far it has really helped improve my understanding of MATLAB and will improve my ability to use the program.

Regards Gavin

[dpb]

Hmmmm....that doesn't seem like should be unless there's something else going on in the input that your sample data file didn't show...

What does

whos U* ix*

show for sizes and

max(U1)
max(U2)

? There should be precisely the number of rows and columns +1 in S as the number of unique values in the two columns and the bin numbers should be 1:1 of bin:element_in_U

As far as understanding, make a very small case that you can see the whole thing on the screen at the command line and work thru the steps and all should become clear...

ADDENDUM:

Oh, looking back I switched up...the first column goes down the rows and the second across columns--reverse the arguments to sub2indx

ix=sub2ind(size(S),ix1+1, ix2+1);  % bin subscripts to linear indices
S(ix)=D(:,3);                      % and store...

And all should be well.

ADDENDUM 2:

Gavin--I just did the section of the file you emailed with the above correction. It looks ok. I didn't catch I had done the reversal earlier as I had used a very small subset that I could see in a larger S array that didn't break the index calculation, sorry.

[Gavin]

dpb,

Thank you this all works perfectly with the getting the data set in the right order? am i missing a step though im still struggling to use the

interp2 function from what i understand i should now define the variables from the new S matrix

lon = S(1,1:end); lat = S(2:end,1); sep= S(2:end,2:end);

and then run the inter2 function to return the separation values at my locations? is this necessary? or is there now a better way of doing it.

thanks again for all your help

Gavin

[dpb]

For the most part if not entirely in the data you provided your missing values appear to be where you would have to extrapolate instead of interpolate to fill in the missing locations.

interp2 presumes the data fill the grid given by X,Y vectors so if you pass in the full range as above (and, btw, lon=S(1,2:end) ) the data array of the missing values is consistent w/ S including the missing values and it will return 0 for those locations except the first that will be the average of the existing value and zero weighted by the two distances to the first missing point.

I'm not full up on what these data really are other than lon,lat--is the sep value really a measured or a calculated value? If it's not computed or you don't have the means of computing it or it is actually measured, about the best I can think of would be to fit a spline to areas of non-missing data in the neighborhood of the boundaries and use it to extrapolate one or two locations then repeat. By the time you get very far away from existing data, however, I'd put virtually no credence in the results unless you know quite a lot about how it must behave and can build that into a model somehow.

[dpb]

For example, below is a subset of the ULH corner of the sample dataset you sent earlier represented as S~=0 to show the locations of available data. As you can see, there's a big void in the upper left triangle that can't be interpolated into with any data whatever on the left/top; similar problems arise frequently throughout.

There are some smaller areas such as the lower rows that do have some surrounding data along the horizontal from which some localized interpolations could be accomplished. It would seem that would likely be better accomplished with interp1 along the given row rather than by interp2, however, since there's still no data above the missing points; only on either side. One could try a two-step process, perhaps.

All in all, the arrangement of values isn't much conducive to easily filling in the missing unless as mentioned earlier you've got auxiliary knowledge to use.

   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
   0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1
   0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1
   0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 1 1
   1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1