OK, starting over -- I looked at your plot and see you really didn't use arrows at all--that makes it much simpler. Why TMW can't have an arrow that works much more easily is beyond me, but...
Start off as did before...presume you've already done the plot and ylim stuff--
Now just repeat for the other set...
So, if you're happy w/ this appearance, you can forget about the annotation stuff. I was trying to make it an arrow pointing down as thought at first glance that's what you'd drawn in your sample.