Why does this piece seem to move when I try to tighten the axle bolt?

Do you mean it tries to back out a wee bit?

When you get your slack set where you want it give your wheel/tire a good smack from the rear toward the front of the bike to set the adjusters. Then tighten the axle nut. You should be good then. You may have to lean your knee against the tire to hold the adjusters snug while tightening the nut. I change wheels a half dozen times a weekend and have to go through this each time as the adjusters are free floating. No biggie.
 
 

When you get your slack set where you want it give your wheel/tire a good smack from the rear toward the front of the bike to set the adjusters. Then tighten the axle nut. You should be good then. You may have to lean your knee against the tire to hold the adjusters snug while tightening the nut. I change wheels a half dozen times a weekend and have to go through this each time as the adjusters are free floating. No biggie. 

Thanks for the help, that's exactly what I ended up doing!

If it bugs you, drill the caps/swingarm and safety wire them together like this (while you're in there put a 3/8 X 1 1/4 spring over the tensioner bolt to hold the adjuster forward while messing with the wheel);
 

 
 
If that doesn't do it for you any of the aftermarket adjusters that lug up to the swingarm would also solve the issue;