Synchronizing 2 Simulink Models on 2 different machines via UDP

Hi, I'm trying to sinchronize 2 simulink models in 2 different machines to divide the work load of each. So far this is a test and my model is a simple mass, damper, spring moving only on the x-asis connected to a PID controller. I change the reference point with a joystick and visualize via VR Sink. Everything works fine if ran in one machine.

So what I did was divide the model so I put the PID controller in one machine, and the Plant with the physics in another machine and connected the models via UDP. If I start the models at the same time (by clicking the run button on both machines at the same time) the model works excellent with what appears to be zero delay.

The problem arises when I start the model in one machine first, and then, few seconds later I start the other. If I do this, the mass behaves like it's on crack and flyes everywhere. I believe this is because there is old data stored in some type of buffer somewhere and the model that starts late gets old data feed to it and it never schincronizes again; however I am not sure of this because in theory UDP is a best effort protocol and old data should be disregarded, right?

Anyways, some more useful information:

Ran on Matlab 2012a.

The model is compiled as a "Real-time windows target" and I use the Analog Input to get data from the joystick and Package output/Input to send/receive UDP data between machines.

From one machine I send the output from the PID controller and the other sends position data from the Physics Plant, so both computers send data to the other.

I did the experiment using a simple wireless router, and also tried Wireless Ad-Hoc connection to make sure the router wasn't causing the problem.

I appreciate any help given. Thanks!

[Jan Houska]

This is not necessarily UDP packet sequence problem. The PID controller is not stateless (has internal states), and if it is started before the plant, it attempts to control the not-yet-running plant. This may set its internal states to values that are not acceptable when the plant is finally connected.

Please see Integral windup for explanation - except that in your case it may be the invalid input value from the not-yet-running plant what is causing the overshooting in the integral term.

[Xenoepist]

I finally fixed my problem, and you are right, if the PID controller starts first, there is still wrong data from the PID controller trying to control something that is not there because the integral part of the PID keeps incrementing. But even so, the instability of my system stayed no matter which model started first. My problem was due to UDP packages getting stored in some type of queue. I will post the solution that worked for me so if anyone has a similar problem, they can try that fix. Thanks for your answer.

[venkadesan]

please post your result i also getting same problem my mail id : [e-mail]

Answer 1

[Xenoepist]

Ok, I found a good workarround my problem. Like Mr. Roberson said, the old packages were stored somewhere in the receiving computer and were not disregarded even thou the program was nor running yet.

So what I did was to consume those packets faster than they were being sent, by changing the sample time on the Packet Input block to be 100 times faster

Answer 2

[Walter Roberson]

Data that manages to make its way between systems, and is matched to an open socket, is not discarded in UDP, not unless the receiving system requests discarding.

UDP packets that are received at a system that has no socket for them should be discarded by that system. But since there is no particular length of time defined for delivery, the packets could in theory be "in transit" for an undetermined amount of time. Your application needs to be able to detect and disregard "old" packets... even in cases where "old" turns out just to be "out of order" (a newer packet was received before an older one.)

[Xenoepist]

First of all, thank you for your help. I am really stuck on this problem.

Now, Ok, so if I understand correctly, one possibility is that there are old packets lingering in the socket. But isn't the socket created by the application? If the application is not running yet, shouldn't there be no socket opened yet? Or could data be stored on the sending socket? uhmmm... but isn't the sending socket just disregarding data after is sent?

On the other hand, if there is no socket open for receiving packages, and there is a direct Ad-Hoc connection between machines, where would the packages be "transiting?" Shouldn't the transit and resending of packages be done by the router? My idea was that if there was no router between the machines, there wouldn't be any packages wondering around in the router.

Again, I appreciate your help. Thanks!

[Walter Roberson]

The socket is created by the application, but if you are using fixed source and destination ports then you need to have a way of determining that packets are not from a previous run of the program.

It doesn't really matter where the old packets get queued: you have a problem no matter where the queuing is occurring.

You need to solve some problems:

1. ensuring that you ignore packets from "old" conversations
2. a mechanism to allow a "new" conversation to be started no matter which program state you are in (because the other end might get reset for example). This mechanism must not rely on single packets in either direction, as those single packets might get lost
3. a mechanism to detect that packets have been received out of order so you can ignore the extra ones
4. position information transfered must not be relative, as relative positioning depends upon every packet getting through
5. udp does not have an inherent timeout, so you need to implement your own mechanism to re-request data or acknowledgment. When you do that, keep in mind that the other end might be sending the information (and re-sending as needed) but the packets might not be getting received (there can be failures that affect only one direction of a link)

Unfortunately as you try to solve these things, you will find that you are nearly re-inventing TCP.

[Xenoepist]

But isn't that too much overhead for a real time aplication? All I want is for the receiving computer to drop all old packages in the queue and just accept new ones... I guess I could send a time stamp with the package and just accept the newer packages and if the packet is too old, just drop it. I'll try that and see how that works.

[Walter Roberson]

When you are using "udp", the receiving computer has no way of knowing which packets are "old" and which packets are "new". udp does not reorder packets in the receive buffer. If packets get out of order somehow, they likely will stay out of order.

Note: a timestamp should be considered to be just a variety of packet sequence number unless the clocks of the two devices are synchronized. Also, watch out for the resolution of the timestamp mechanism: depending exactly which mechanism you use to timestamp, you might get the same timestamp for several packets in quick succession (the computer often runs faster than the timestamp updates.)

[Xenoepist]

Actually my plan was to sinchronize the computers (somehow) as close as possible and use a microsecond time stamp using S-functions to disregard packages more than 1 millisecond old. However, I am not too savy with c and I have not found any way to get this much time resolution and even if I do, I am not sure it will work with Real Time Windows Target, and even if I can do all of this, I don't think I can sinchronize two computers down to the milliseconds, but it was worth a shot.

But now that I think of it If I am able to get the milliseconds stamp, I could send it from one computer and receive it back as a hand shake to start a conversation. That way if the timestamp received is close to the actual time stamp, the program has green light to start a conversation. I know this doesn't take care of packages out of order, but so far my only problem has been the piling of old packets due to one simulation starting earlier than the other. once in sinch, I don't think there will be much problem, specially because both computers are connected directly.

[Xenoepist]

After I resolved my problem I noticed that I had little or no clue of what was going on under the hood of simulink so I tried a couple of experiments to see if I could understand the issue at its heart.

I did figure out a couple of things. 1) There is a buffer of 16384 bytes that stores the first packets that enter it... I really think the real size is 8192 (2^13) bytes and that I did something wrong with my experiment, but 16384 is the number that I got (2^14). 2) This buffer starts saving data as soon as I click the "Connect To Model" button on the receiving end. I am not sure, but I think that this buffer has nothing to do with the UDP protocol and everything to do with the Packet Input box of simulink. I wish I knew how to reset it. Thanks for your answers, thou, they helped me out a great lot.

[Walter Roberson]

https://en.wikipedia.org/wiki/Precision_Time_Protocol PTP is a possible option for synchronizing local computers.

Answer 3

[K E]

A workaround to your original problem might be to run your Simulink model with the Parallel Computing toolbox one just one machine so that Simulink runs faster, but you do not need both machines. Here are some tips on faster Simulink execution. If I have misunderstood your problem, sorry.

[Xenoepist]

Actually, the whole purpose of the experiment is to make it able to work on two computers. It is actually a school project that involves 2 computers, Blender, and Real Time Windows Target. But thanks for helping me out. I did find an answer to my problem, and I am about to post it if it helps anyone having the same problem.

[saiful azrie]

i did the same project, but i cannot find the problem like u before. can i know how to perform the problem?

[Sree]

@Xenoepist, Could you please let me know how you have solved the problem?