There is also the theory which involves Mercury as an ex-moon of
Venus, largely based on calculations done by Van Flandern and Harrington (A Dynamic Investigation of the Conjecture that Mercury is an Escaped Satellite of Venus. Icarus 28: 435-40 (Abstract), 1976) and goes as follows (Van Flandern, Missing Planets, Dark Matter, and New Comets, 1999):
As Mercury tidally drifted outward it necessarily produced rotational
drag on Venus, and it raised even bigger tides on the Venusian
atmosphere causing it to circulate in retrograde direction. After
billions of years this might impart retrograde motion on the whole
planet.
Tides caused on Venus by Mercury while the latter was still spinning
rapidly would have caused great interior heating and outgassing, and
probably a great deal of surface upheaval (mountain building), too,
causing the very dense atmosphere, the massive release of carbonate in
the rocks as CO2 into the atmosphere, and the very high mountains.
Mercury is massive enough to have taken much of Venus's spin in the
1st half-billion years after formation and Venus's orbit is close
enough to the Sun that complete escape occurs. The interchange of
energy between Venus and Mercury would have been enormous, given
Mercury's large mass (4 1/2 times more massive than the Moon).
Most of the iron (which eventually produces the magnetic field) in
Venus would have been forced up into the crust by an excessively high
spin rate, with Mercury getting most of the iron during fissioning,
which would explain why Mercury has a stronger magnetic field than
Venus. By contrast, the Earth's iron was not forced to the surface,
perhaps because the Earth was not as hot and molten as Venus during
that phase of its formation.
During its lunar phase Mercury would have acquired a prolate shape
(somewhat elongated towards Venus) because of tidal forces.
Both planets would have been melted by tidal heating in the early
stages following escape. If this occured before Venus differentiated,
it might have caused Mercury's high density and stronger magnetic
field. Subsequently, both planets would have melted from mutual tidal
heating.
After escape, Mercury acquired greater tilt and eccentricity, and
Venus would have lost more of its spin. Its prolate shape would have
been reduced after escape but still maintained.
At the point of escape Mercury would have had a period of revolution
of about 40 days, and would have retained its spin period, which would
also be 40 days since it was locked with Venus. But tides raised by
the Sun would slow down its spin to its present 60 days, which gives
it a 3-2 spin-revolution ratio (3 spins per 2 revolutions, in other
words, its rotational period is 2/3 its period of revolution, which is
88 days), because the next stable configuration for such a body
(Mercury mass and diameter and degree of prolateness) is this ratio,
so it is a predicted outcome of its having been a moon of Venus.
This model, then, explains all the anomalies of both Venus and Mercury. Musser (2006) says it would require too much time for Venus to lose a moon but does not provide any reference for this, and the possibility has been corroborated by Kumar (1977) and Donnison (1978). This is the abstract from Donnison:
Kumar's (1977) suggestion that the slow rotations of Mercury and Venus are in part due to natural satellites that subsequently escaped is discussed. A more useful criterion for the escape of such satellites than that previously proposed is derived, and it is shown that this distance is sufficiently small for Mercury and Venus to make the escape of satellites a likely possibility.
And this is the abstract from Kumar:
It is suggested that the slow rotations of Mercury and Venus may be connected with the absence of natural satellites around them. If Mercury or Venus possessed a satellite at the time of formation, the tidal evolution would have caused the satellite to recede. At a sufficiently large distance from the
planet, the sun's gravitational influence makes the satellite orbit unstable. The natural satellites of Mercury and Venus might have escaped as a consequence of this instability.
They do not, however, specifically say that Mercury was once a moon of Venus.
This is the abstract from Van Flandern and Harrington (gizidda.altervista.org):
The possibility that Mercury might once have been a satellite of Venus, suggested by a number of anomalies, is investigated by a series of numerical computer experiments. Tidal interaction between Mercury and Venus would result in the escape of Mercury into a solar orbit. Only two escape orbits are possible, one exterior and one interior to the Venus orbit. For the interior orbit, subsequent encounters are sufficiently distant to avoid recapture or large perturbations. The perihelion distance of Mercury tends to decrease, while the orientation of perihelion librates for the first few thousand revolutions. If dynamical evolution or nonconservative forces were large enough in the early solar system, the present semimajor axes could have resulted. The theoretical minimum quadrupole moment of the inclined rotating Sun would rotate the orbital planes out of coplanarity. Secular perturbations by the other planets would evolve the eccentricity and inclination of Mercury's orbit through a range of possible configurations, including the present orbit. Thus the conjecture that Mercury is an escaped satellite of Venus remains viable, and is rendered more attractive by our failure to disprove it dynamically.