My question is, then, how does the Moon's orbit manage to not have any concavity, no matter how minute, while transitioning from the full moon to the new moon positions? Isn't that not a mathematical impossibility?
TL;DR answer: Because the gravitational acceleration of the Moon toward the Sun is about twice the gravitational acceleration of the Moon toward the Earth and because the speed at which the Earth orbits the Sun is about thirty times the speed at which the Moon orbits the Earth.
To get the longer answer, a definition of what "convexity" means is needed. This is easy for a simple closed plane curve: A simple closed plane curve (aka a Jordan curve) is convex if for any two points on the interior of the curve, all of the points on the line segment connecting the two points lie in the interior of the curve.
Unfortunately, neither the Earth's nor the Moon's path about the Sun is closed or planar. To resolve these issues, I'll first do what the referenced paper did, which was to instead investigate coplanar circular orbits of a point mass planet about a star and of a infinitesimal point mass moon about the planet, such that
- The planet's orbit velocity $v_p$ about the star is inversely proportional to the square root of the distance $r_p$ between the star and the planet,
- The moon's orbit velocity $v_m$ about the planet is inversely proportional to the square root of the distance $r_m$ between the planet and the moon, and
- The constant of proportionality for the planet $\left(\frac{{v_p}^2}{r_p}\right)$ is much greater than that for the moon $\left(\frac{{v_m}^2}{r_m}\right)$.
For sufficiently small values of $r_m$, the moon's orbital velocity about the planet will exceed that of the planet about the star, making the moon's path about the star intersect itself:
The loops become small as the moon's orbital distance increases, eventually becoming inward pointing cusps at the point where the moon's orbital velocity about the planet has decreased to where it equals the planet's orbital velocity about the sun. While this curve might or might not be closed, it definitely is not convex due to the inward facing cusps. The cusps broaden into intervals where the primary normal points outward as the moon's orbital radius is increased even further:
This curve is still non-convex, as exhibited by how the primary normal alternates between pointing inward and outward. the curvature of the moon's path about the star is zero at these transition points: The path, at least instantaneously, is a straight line. This happens because the acceleration vector is parallel to the velocity vector at those transition points.
The intervals where the moon's path about the sun is concave rather than convex shrink as the moon's orbital distance increases even further. At some critical point the intervals of concavity shrink to nothingness. The path is convex everywhere at this orbital distance and beyond:
These critical points occur where the acceleration of the moon toward the star is identically zero. This should not be surprising as there is a very close connection between velocity, acceleration, and curvature. In particular, the curvature of a curve at some point is
$$\frac{d\hat T}{ds} = \frac{(\vec v \times \vec a)\times \vec v}{v^4} = \kappa \hat N$$
where $\hat T$ is the unit tangent, $\vec v$ are the velocity and acceleration of a point that follows the curve over time, $\kappa$ is the curvature (the inverse of the radius of curvature), and $\hat N$ is the primary normal to the curve. A simpler expression for the curvature is
$$\kappa = \frac{||\vec v \times \vec a||}{v^3}$$
Note that the vector $\vec v \times \vec a$ points in the direction of the unit binormal. This suggests a simple metric that extends to non-planar orbits: An orbit about some central point if the cross product between velocity and acceleration with respect to that central point always lies in the same half plane. An even easier metric is to test whether the magnitude of the gravitational acceleration toward the star is greater than that toward the planet.