Regarding solar system dynamics, i.e. planets in stellar systems and moons in planetary systems, this is often mentioned in the literature, but it is difficult to find a good analysis/explanation of this phenomenon.
$\begingroup$ Clarification would help. If you mean rotation direction versus orbital direction for planets, I understand this is a standing problem in current models of solar system formation, as fluid mechanics give us predictions that don't match observation. Standard models propose that planetary accretion began as vortices in the fluid-like planetary disc, and planets got their prograde rotation there. But if you apply fluid mechanics then you find that retrograde vortices are significantly more stable than prograde ones. $\endgroup$– zibadawa timmyJul 14, 2015 at 11:21
In any system of orbiting bodies, the length of time two bodies spend in proximity determines the mutual gravitational perturbation of each body on the other.
If both bodies are orbiting in the same direction, the inner body (closer to the central mass [primary star, for instance]) will overtake the outer one, because it is orbiting at a faster angular velocity. Because both bodies are orbiting in the same direction, they will have a great deal of time close to one another, and more time mutually to disturb one another. This is especially pronounced when the two orbits are adjacent and close.
On the other hand, if two bodies are orbiting in opposite directions, they spend very little time in close proximity, so gravitational perturbation is minimized. Passing one another happens very quickly.
By analogy, compare the amount of time two vehicles spend close to one another on a highway when one is passing the other and going in the same direction. A passenger in one could have time to study the other vehicle in great detail. Then, compare two vehicles moving in opposite directions; any passenger could only gain only a brief glimpse of the other vehicle up close. The amount of time for a passenger to study the other vehicle is analogous to the amount of gravitational disturbance one orbiting body will have on another.
A similar problem occurs when the orbital periods of two bodies are simple ratios of one another (orbital resonance). The repeated tugging of both bodies on one another at the same place in their orbits creates an additive distortion at that location in their orbits. This problem would occur in both situations -- orbiting in the same direction and opposing directions.
Though the reasoning is simple, the exact amount of perturbation is quite complex. A "simple" 3-body problem requires the use of advanced mathematics (celestial mechanics).
$\begingroup$ Wow what a great explanation. Thanks for that. $\endgroup$– FattieJul 3, 2016 at 14:34
$\begingroup$ While the retrograde objects spend less time passing each other, they pass each other more frequently. I would think overall, the effect should be pretty similar. $\endgroup$– userLTKJul 4, 2016 at 8:54