I need to calculate the relative angular path between two planets in the geocentric view or in other words the number of degrees of a circle one planet travels away from another planet around the Earth. In the heliocentric system I simply subtract the longitude of one planet from the longitude of the other and add n*360 if they made full cycles in between. Would the same work for the geocentric coordinates? Does the loop a retrograding planet makes in the sky actually count as its path? Should I consider the retrograde movement as an optical illusion or include it in the formula and how?
If you are calculating the angular distance between planets, as viewed from Earth, then of course you need to include the retrograde motion.
The natural way of doing this would be to first use heliocentric coordinates to calculate the positions of both planets and the Earth relative to the sun (You could do this using Kepler's laws). Then change your coordinate system to the calculated position of the Earth.
Since you are using a calculated position of the Earth, the apparent retrograde motion (which is caused by the movement of the Earth) is taken care of, and doesn't need any special handling.
When you have found the position of the planets, you can find the angle between them by straightforward spherical geometry. (Since the planet's don't all orbit in exactly the same plane, just subtracting longitudes is less accurate, but that is the same whether you are working geo- or helio-centrically.)
It's possible, but much harder, to work in geocentric coordinates throughout. A good approximation can be found using Ptolemy's work. However Keplerian ellipses are more accurate and simpler.
In our moving Earth-based point of view, retrograde apparent motion means a planet's geocentric ecliptic longitude is temporarily decreasing instead of increasing as usual. The word "apparent" avoids confusion with a retrograde orbit. All major planets' orbits are prograde, with heliocentric longitude increasing at all times1. Most retrograde-orbiting Solar system objects are comets.
As seen from Earth, the apparent angular separation between two planets near the ecliptic is approximately2 the difference between their geocentric longitudes at that point in time, regardless of their rates of change in longitude. Even if you plot this value versus time, your calculation need not break apparent motion down into positive and negative cases.
- Except where longitude wraps around from 360$^\circ$ to 0$^\circ$.
- When longitude difference is small (near conjunction), latitude difference is also significant.