How to deal with repeated data in a column of a table

Hey,

I'm currently trying to process a table. What I want is to extract a sequence from a column of it. Before I used unique but apparently the result wasn't what I intended for.

An example of the table could be like below. I want to extract [22 24 22] for vid 59. However, unique only gave me [22 24], which is not what I want. So, I hope I've explained my question well. Appreciate your advice on this!!

Answer 1

[Cyrus Tirband]

It sounds like you're looking to remove repeating elements that are adjecent. If that is the case, you can test your array for differences between each element and remove the parts where the difference is 0.

If your array is named A, then:

A(diff(A)==0) = []

[QIAO WANG]

Thank you. I tried this way but it doesn't work. In fact, what is this method diff? I can't find its detailed explanation.

[Cyrus Tirband]

diff takes the finite difference. If B = diff(A) then B(i) = A(i+1)-A(i);

What is the type of the variable where you store your data? Is it an array?

In the following example, it works:

A = [22; 22; 22; 24; 24; 24; 22; 22; 22];
A(diff(A)==0) = []

A =

    22
    24
    22

[QIAO WANG]

Well, something weird happened to me. When I executed the code, error occured.

>> A = [22; 22; 22; 24; 24; 24; 22; 22; 22];
A(diff(A)==0) = []
Index exceeds array bounds.

That's why I said it didn't work to me. Why is this? Is it because of my Matlab version?

Regarding my data type, it's in a table as I showed. Essentially, I want to process the second column, which is basically a vector. So, in my opinion, if I can find a way solving it in a vector format, my problem is gonna be resolved.

So, I think your idea is really nice and seems to work on your side. But why is it not working on my side?

[Cyrus Tirband]

Strange. In the example with

A = [22; 22; 22; 24; 24; 24; 22; 22; 22];

Can you display the following:

diff(A)
diff(A)==0

So that we can have an idea of what's going on?

[QIAO WANG]

Hey, thanks. I found the problem. I got a variable called diff so there was some conflict...

So, now I'm gonna have a look at this and see if I can use it to deal with my table. Hopefully I won't turn back to you for help again. Thank you so much.

[QIAO WANG]

Hey Cyrus.

I've got another data processing problem. So could you please have a look?

As I mentioned earlier, I'm dealing with a table and it looks like this. I've shortened it a bit to show here. In my last question, I'd like to extract 19 22 24 22 24, which was successful thanks to your help and that was for a different purpose. Now, based on Table I, I'm going to get the first row when a BSid first appeared and the last row when the same BSid lastly appeared. For example, I'd like to obtain a table which looks like Table II. I've been thinking about this for a while but unfortunately, no good idea has come into my mind.

So, I wish you could have some good idea about this and thank you in advance.

Table I:
     ID      Time         BSid
     59      08:47:31     19 
     59      08:47:32     19 
     59      08:47:33     19 
     59      08:47:34     19 
     59      08:47:35     22 
     59      08:47:36     22 
     59      08:47:37     22 
     59      08:47:38     22 
     59      08:47:39     22  
     59      08:47:45     24 
     59      08:47:46     24 
     59      08:47:47     24 
     59      08:47:48     24 
     59      08:47:49     24 
     59      08:47:50     22 
     59      08:47:51     22 
     59      08:47:52     22 
     59      08:48:00     22 
     59      08:48:01     24 
     59      08:48:02     24 
     59      08:48:03     24 
     59      08:48:04     24 
     59      08:48:05     24 

Table II:
     ID      Time         BSid
     59      08:47:31     19 
     59      08:47:34     19 
     59      08:47:35     22 
     59      08:47:39     22  
     59      08:47:45     24 
     59      08:47:49     24 
     59      08:47:50     22 
     59      08:48:00     22 
     59      08:48:01     24 
     59      08:48:05     24

[Cyrus Tirband]

Hi Qiao,

You can use a similar approach. In this case, you have two requirements for the elements that you want to remove. Not only must the finite difference diff(A) be equal to zero (no repeating elements), the second finite difference, diff(diff(A)) must also be equal to zero (no change in slope). So, and(diff(A) ==0, diff(diff(A)) == 0) is the set that must be removed. Using de morgan's law this can be written a bit more succinctly as ~or(diff(A), diff(diff(A))).

Of course, you always want the first index to be included, so you need to enter that one manually as a 1, since diff(diff(A)) will be one index shorter.

In code:

A = [22; 22; 22; 24; 24; 24; 22; 22; 22];
A(~or(diff(A),[1; diff(diff(A))])) = []

A =

    22
    22
    24
    24
    22
    22

[QIAO WANG]

Thank you Cyrus. Actually, before you replied to me, I also raised a question in the community and got a good one. Anyway, yours is a brilliant idea! Worked for me too.