I don't know how to calculate the minimum possible forbidden region for a planet of a specific mass orbiting a star of a specific mass at a specific semi-major axis. So I can't say what the absolute theoretical minimum separation between the orbits of two planets in the same planetary system can be.
A reductio ad absurdum calculation suggests that the minimum possible separation between planetary orbits could be about 2,500 to 5,000 kilometers. Probably astrophysicists should be able to calculate a minimum possible separation of orbits which would be many, many times that.
There are also some theoretical configurations of planets which would result in two or more planets sharing the same orbit, in which case one could be facetious and claim there is a zero separation between their orbits.
More Detailed Answer:
Part One of Six: Some "Dole ful" Calculations.
I do know that there have been formulas to calculate the minimum possible spacing between planets in a star system for decades.
Habitable Planets for Man, 1964. 2007, by Stephen H. Dole, is a scientific discussion of the parameters of planets which are habitable for humans.
Chapter Three: introduction to General Planetology, has a section "Spacing of Planets in the Solar System", on pages 49 to 52, discusses the spacing of planets in our solar system.
Dole bases his discussion of forbidden regions on his own paper:
"Limits for stable Near-circular Planetary or Satellite Orbits in the Restricted Three-Body Problem". ARS J, 31, no. 2 (February, 1961), pp. 214-219.
Dole calculated the limits of the "forbidden" regions of all the planets in the solar system, based on the masses of each planet and the Sun, the eccentricity of each planet's orbit, and the semi-major axis of each planet's orbit around the Sun.
The "forbidden" region is a ring around the Sun that a planet orbits within and which no other planet should be able to have a stable orbit within.
According to Dole's calculations, our solar system out to Neptune or Pluto is approximately half full of forbidden regions around planetary orbits:
This pattern of regularity should also be found in other planetary systems. Forbidden regions take up about 50 per cent of our solar system, and if this is true of other planetary systems (or multiple star systems); then it would be a simple matter to design any number of stable planetary systems by random mechanical processes.
So, according to Dole's calculations, a star like the Sun might have the same number and masses of planets as the Sun does, if its planetary system was about 50 per cent filled by planets and their forbidden regions. If planets could be spaced close enough together that the limits of their forbidden regions touched, then our solar system could have perhaps twice as many planets with the same masses, about 16 to 18.
More importantly to me, if there are one, two, or three planets within the circumstellar habitable zone of the Sun, in the planetary system of a star like the Sun with the forbidden zones of the planets just touching, there could be only two to six planetary orbits within the circumstellar habitable zone of that star. As a kid who liked science fiction stories where many planets in a single solar system were habitable, I found that idea rather "Dole ful".
Part Two: Some Early Ideas to Increase the Number of Habitable Planets in a system.
A number of science fiction stories have habitable planets in Trojan orbits, in which a planet orbits around a star in the L4 or L5 Lagrange point of another planet or another star, 60 degrees ahead of or behind the other astronomical object.
Such Trojan orbits are known to work when the primary, secondary, and tertiary objects differ in mass by hundreds or thousands of times. In our solar system, the Sun is thousands of times as massive as planets which are thousands of times as massive as the asteroids in their Trojan positions. And Saturn is thousands of times as massive as its moons Tethys and Dione, which in turn are thousands of times as massive as their Trojan moons.
But could two planets in the size range to be habitable share the same orbit, one planet being in the L4 point of the second one, which would be in the L5 point of the first one? I don't know.
But if that was possible, and if there can be two to six planetary orbits within the circumstellar habitable zone of a star like the Sun, then if there are two Trojan planets in each orbit that solar system could have four to twelve potentially habitable planets in its circumstellar habitable zone.
Another way to increase the number of habitable planets within the circumstellar habitable zone would be to replace each Earth mass habitable planet with a double planet having the same total mass as the Earth. According to Dole, the minimum mass for a habitable planet would be about 0.4 Earth mass, so a double planet with twin planets of about 0.42 Earth mass would have a total mass of 0.84 Earth, and thus a slightly smaller forbidden region than Earth.
So theoretically about four to twelve habitable planets could orbit in the circumstellar habitable zone of a star identical to the Sun, if arranged as two to six sets of twin planets.
Part Three: Co-Orbital Planets?
In astronomy, a co-orbital configuration is a configuration of two or more astronomical objects (such as asteroids, moons, or planets) orbiting at the same, or very similar, distance from their primary, i.e. they are in a 1:1 mean-motion resonance. (or 1:−1 if orbiting in opposite directions).1
There are several classes of co-orbital objects, depending on their point of libration. The most common and best-known class is the trojan, which librates around one of the two stable Lagrangian points (Trojan points), L4 and L5, 60° ahead of and behind the larger body respectively. Another class is the horseshoe orbit, in which objects librate around 180° from the larger body. Objects librating around 0° are called quasi-satellites.
An exchange orbit occurs when two co-orbital objects are of similar masses and thus exert a non-negligible influence on each other. The objects can exchange semi-major axes or eccentricities when they approach each other.
The space probes Pioneer 11, Voyager 1, and Voyager 2 discovered several new moons of Saturn when they passed it in 1979, 1980, and 1981. This included the discovery of the tiny moons Epimetheus and Janus, which are co-orbital.
The Saturnian moons Janus and Epimetheus share their orbits, the difference in semi-major axes being less than either's mean diameter. This means the moon with the smaller semi-major axis will slowly catch up with the other. As it does this, the moons gravitationally tug at each other, increasing the semi-major axis of the moon that has caught up and decreasing that of the other. This reverses their relative positions proportionally to their masses and causes this process to begin anew with the moons' roles reversed. In other words, they effectively swap orbits, ultimately oscillating both about their mass-weighted mean orbit.
So it would potentially be possible for two habitable planets to share the same orbit in an exchange orbit like that of Epimetheus and Janus, thus having two habitable planets in the orbit and forbidden region. That could be used to double the number of planets orbiting within the habitable zone of a star.
I can't help thinking that inhabitants of a planet in an exchange orbit with another planet would find the exchange process terrifying until and unless they could calculate that the two planets were not going to collide.
Part Four: Exoplanet Discoveries.
IN the last generation thousands of exoplanets have been discovered orbiting other stars. And planetary systems with two or more planets orbiting the same star have also been discovered. Because of the great difficulty in discovering exoplanets, it is reasonable to assume that in most cases there are more planets in a system than have been discovered yet, perhaps more planets than can be discovered until decades, centuries, or even millennia of future scientific progress is made.
Even though Dole wrote in 1964 that:
This pattern of regularity should also be found in other planetary systems.
The majority of planetary systems discovered around other stars have been significantly different in one or more major ways from our solar system. Thus there must be a great variation in the processes which form and shape planetary systems.
According to Wikipedia's list of exoplanet extremes, the smallest distance between the semi-major axis of the orbits of two consecutive planets is between Kepler-70b and Kepler-70c, about 0.0016 AU, or about 240,000 kilometers, or about 149,129 miles, closer than the distance between the Earth and the Moon.
If each consecutive planetary orbit around the Sun was only 0.0016 AU farther out than the previous one, 262 planetary orbits could fit within Kasting's conservative habitable zone for the Sun, and 518 in his optimistic habitable zone.
However, the article on Kepler-70 indicates that it is now believed the planets do not exist and their detection was probably an error.
I do not know what is the smallest known difference between the orbits of two confirmed exoplanets. But the two exoplanets with the smallest ratio between their orbits are Kepler-37b & Kepler-37c. Both planets are several times as massive as Earth but orbit very close to their star Kepler-37 and thus to each other.
The orbit of Kepler-37b has a semi-major axis of 0.1153 AU, or 17,248,634 kilometers, or 10,747,804.57 miles, and the orbit of Kepler-37c has a semi-major axis of 0.1283 AU, or 19,193,407 kilometers, or 11,926,230 miles, a difference of 0.013 AU, or 1,944,772.3 kilometers, or 1,208,425.49 miles.
If the planetary orbits around the Sun could have an average separation of 0.013 AU, 32 planetary orbits could fit within Kasting's conservative habitable zone and 64 planetary orbits could fit within Kasting's optimistic habitable zone.
0.1283 AU is 1.1127 times 0.1153 AU, and is the smallest known ratio between consecutive planetary orbits. If I remember correctly I calculated that 4 planetary orbits with that ratio could fit within Kasting's conservative habitable zone and 6 planetary orbits could fit within Kastings optimistic habitable zone.
So should the distance of 0.013 AU between the orbits of Kepler-37b and Kepler-37b be considered the minimum possible spacing between consecutive planetary orbits?
Or should the ratio of 1.1127 between their semi-major axis be considered to the minimum possible ratio between consecutive planetary orbits?
If it is the ratio between orbits which determines the minimum spacing of planetary orbits, if a planet orbits very close to its star the minimum possible distance to the next planet's orbit could be less than the 0.013 AU between the Kepler-37 planets. For example, if the inner planet orbits at 0.01 AU, the next planet could orbit at a distance of only 0.011127 AU, a difference of only 0.001127 AU, a little less than the distance between the orbits of the alleged Kepler-70 planets.
If it is the ratio between orbits which determines the minimum spacing of planetary orbits, if a planet orbits very far from its star the minimum possible distance to the next planet's orbit could be many times the 0.013 AU between the Kepler-37 planets. For example, if a planet orbits 100 AU from its star, and the minimum separation is 1.1127 times the inner orbit, the next planet out would have to orbit at a distance of at least 111.27 AU.
Par Five: Planets in Rings.
Astrophysicist Sean Raymond, in his PlanetPlanet blog, has a section called Ultimate Solar System designing planetary systems with as many planets in the habitable zone as he can fit in.
In the Ultimate Engineered Solar system Raymond designs a planetary system with 416 planets in the habitable zone, using rings of planets sharing the same orbit.
That is based on this paper by Smith and Lissauer:
Smith and Lissauer calculate that seven to forty two planets can share the same orbit around their star, if the planets have equal mass and are equally spaced around the star.
The planets would be separated by 8.57 degrees if there are forty two planets in the ring, increasing as the number of planets decreases, so that seven planets would have gaps of 54.42 degrees between them. The size of those gaps in AU, kilometers, or miles would have to be calculated from the semi-major axis, and thus the circumference, of the orbit.
Raymond also states that planetary orbits should be separated by 5 to 10 times their Hill radius to be stable. For example, a planet with the mass of Earth 1 AU from a star with the mass of the Sun would have a Hill radius of about 1,500,000 kilometers or 0.01 AU.
So planetary orbits of Earth-like planets orbiting Sun-like stars at distances of about 1 AU should be separated by at least 0.05 to 0.10 AU or 7,500,000 to 15,000,000 kilometers.
Part Six: Conclusion.
The minimum possible distance between the semi-major axis of the orbit of a planet and the semi-major axis of the the orbit of another planet orbiting the same star can be calculated from factors such as the mass of the planet and the mass of the star, and the semi-major axis of the planet's orbit.
Known examples of the separation of the semi-major axis of consecutive planets in a planetary system range from as much as about 662 AU between CVSO 30 b & CVSO 30 c down to as little as 0.013 AU between Kepler-37b & Kepler-37c.
Known examples of the ratios of the semi-major axis of consecutive planets in a planetary system range from as much as about 78,998 times the distance in the case of CVSO 30 b & CVSO 30 c down to as little as 1.1127 times the distance in the case of Kepler-37b & Kepler-37c.
Of course if additional planets are discovered between the orbits of the known planets in those systems the records could change.
Astronomers are fairly certain that any planetary orbital separation within those ranges is possible. I don't know the theoretical limits to how much larger or smaller the separation between the semi-major axis of two consecutive planetary orbits could be.
For minimum spacing I guess I can do a reductio ad absurdum calculation.
According to Sean Raymond here
The minimum possible separation between two planetary orbits should be five to ten times the Hill sphere radius of one of the planets - the planet with the larger Hill sphere radius.
The absolutely smallest the Hill radius of a planet orbiting very close to a very massive star could get would be at the surface of that planet. If the surface of the planet extended above the Hill sphere radius the surface material would be ripped from the planet by the gravity of the more massive object until the planet was stripped down to to the radius of the Hill Sphere.
If a astronomical object has to be pulled by its gravity into a spheroidal shape to be considered a planet, that establishes a minimum size for a planet or a planemo (planetary mass object), and thus for its smallest possible Hill sphere. The minimum radius necessary for astronomical object to be spheroidal is not known precisely.
More or less arbitarily making a radius of 500 kilometers the lower limit for a planetary mass object, and assuming that such an object could orbit close enough to its star to have it's hill sphere at its surface, the minimum possible separation between its orbit and that of the next planet in the system would be about 2,500 to 5,000 kilometers.
And of course if two planets shared the same orbit, such as Trojan planets, planets in an exchange orbit, or planets in a ring, someone could facetiously say there would be approximately zero difference between the semi-major axis of their orbits, since their orbits could be the same orbit.