A planet can get tidally locked to its moon, like Pluto and Charon, or like Earth someday and Luna. A planet can get tidally locked to its sun - close orbiting exoplanets are assumed to be tidally locked to their stars.
If a close orbiting planet had a substantial moon, could the presence of the moon prevent the planet from becoming tidally locked to its sun? The planet cannot be locked to both. Or does the star win in this contest of who can tidally lock the planet?
Yes, but the tidal torques will prevent the planet from being locked either to the star or moon.
Suppose the orbital paths are both prograde and coplanar and the planet obliquity is zero. Then tidal torque exerted on the planet by its moon should be:
$$N_m=\frac{9}{4}k\frac{Gm_m^2A^5}{r_m^6}\sin{(2\alpha_m)}$$
Here, $m_m$ is the moon mass, $G$ is the standard gravitational parameter, $A$ is the planetary radius, $r_m$ is the distance between the moon and planet, $\alpha_m$ is the tidal lag angle, and $k$ is the quadrupole tidal Love number of the planet. Note that if the planet is already tidally locked to the moon, then $\alpha_m = 0$, so there is no torque being exerted on the planet.
Similarly, the tidal torque exerted on the planet by the star is: $$N_s=\frac{9}{4}k\frac{Gm_s^2A^5}{r_s^6}\sin{(2\alpha_s)}$$
Here, $m_s$ is the star mass, $r_s$ is the distance between the star and planet, and $\alpha_s$ is the tidal lag angle.
The moon's orbital period around the planet will be much smaller than the planet's orbital period around the star. So $\alpha_s \ne 0$. This means there will be significant torque on the planet in the opposite direction of its rotation, due to the star. As this torque slows the rotational speed of the planet, the planet will move out of tidal lock with the moon. This will increase the absolute value of $\alpha_m$, and the moon will begin exerting tidal torque on the planet in the direction of its rotation.
Then $N_m$ and $N_s$ will be rotation torques in opposition. For some rotation rate for the planet, they will be equal and opposite, and the planet's rotation rate will stabilize, without it being tidally locked to either its moon or star. Venus is in a similar situation with the Sun. The tidal torque on Venus due to the Sun is in opposition to the torque on Venus due to atmospheric thermal tides, which prevents it from being tidally locked.
Notes:
Tidal torques conserve angular momentum, so rotational momentum is transferred into orbital momentum, causing planets' and moons' orbits to migrate. If a moon loses too much orbital momentum it will crash into the planet. If it gains too much, it will exit the planet's hill sphere and be ejected (or crash back into another body). See Alavarado-Montes and Sucerquia 2019. For close-in planets, tidal torque is much higher than for far-out planets, scaling as $1/r^6$. This could be why we have observed no exo-moons on the abundant close-in, hot-Jovian exoplanets.
The above answer only considers planetary torques cause by the moon's gravity on the moon generated tidal bulge, and the star's gravity on the star generated tidal bulge. It doesn't take into account the planetary torques due to each body on the other body's tidal bulge. That is because these cross-bodied torques will pass through every lag angle each time the planet orbits the star. However, certain resonances in lunar and stellar rotation rates could change quickly change orbital parameters and destabilize the 3-body system.
This is discussed in Habitable Planets for Man, Stephen H. Dole, 1964.
https://www.rand.org/content/dam/rand/pubs/commercial_books/2007/RAND_CB179-1.pdf
Starting on page 67 Dole discusses the upper mass limits for the star of a habitable planet. Above a certain mass, a star will not stay on the main sequence long enough for a planet to develop a brathable atmosphere. Dole then discusses the lower limits of mass for a star which can have a human habitable planet.
The less massive a main sequence star is, the less luminous it is, and so a habitable world with habitable temperatures would have to orbit closer. Below a certain stellar mass the luminosity is so low that a planet orbiting in the circumstellar habitable zone (which Dole calls the "ecosphere") would suffer such strong tidal forces from the star that it would be tidally locked.
Dole discusses the lower limits of mass due to tidal braking forces on pages 68 to 72. On pages 71 to 72, Dole writes:
As may be seen from figure 26, if h2 equal to 2.0 is used as a criterion, habitable planets can exist in ecospheres only around stars having masses larger than 0.72 solar mass. A "full" ecosphere can exist around primaries of stella rmass greater than about 0.88 solar mass, but the ecosphere is narrowed by the tidal braking effect for primaries of lesser mass until it disappears when the stellar mass reaches about 0.72.
The range in mass of stars that could have habitable planets is thus 0.72 to 1.43 solar masses, corresponding cto main sequence stars of spectral classes F2 to K1. There is an extension of this range down to the larger class M stars (mass greater than 0.35 solar mass) for a special class of planets with large satellites. This will be discussed in the next section.
According to this chart, a class F3V star has a mass of 1.44 solar masses:
https://en.wikipedia.org/wiki/F-type_main-sequence_star
According to the chart here a K0V type star has a mass of 0.88 solar masses:
https://en.wikipedia.org/wiki/K-type_main-sequence_star
This may reflect more accurate knowledge of the masses and spectral types of stars since Dole wrote.
In the next section Dole writes on pages 72 & 73:
The tidal barking forces of a primary also apply to satellites. If h2 of a satellite on a planet is greater than 2.0 but that of the primary is less than 2.0, one would expect to find the planet's rotation halted with respect to the satellite but continuing with respect to the primary. The planet's solar day and synodic month would thus be of the same length. For this condition to be compatible with habitability, however, the period would have to be such as to produce a solar day less than 96 hours in duration--a figure rather arbitrarily chosen as thelongest day for habitability.
And Dole goes on to discuss calculating the tidal forces of the satellite and the primary on a planet.
On page 74 Dole writes:
It is interesting to note that within a certain range of satellite masses twin habitable planets that did not rotate with respect to each other could exist.
So according to Dole the satellite could be as large as the planet. And he goes o to write:
The right hand portion of figure 28 rpesents the situation in which the satellite is larger than the habitable planet.
Which of course makes the habitable "planet" really the satellite or moon, and the more massive "satellite" actually the planet.
In recent decades there has been much discussion of the possibility that very large exomoons orbiitng giant exoplanets that orbit in the circumstellar habitable zones of their stars could be habitable. Since the majority of stars are red dwarfs where planets in the habitable zones would be tidally locked, which may be inconsistent with being habitable, giant moons orbiting giant planets in the habitable zones might be habitable, since they would be tidally locked to their planets instead of to their stars.
Heller and Barnes discuss whether such exomoons would be tidally locked to their planets or to their stars in "Exomoon habitability constrained by illumination and tidal heating", 2013.
https://arxiv.org/vc/arxiv/papers/1209/1209.5323v2.pdf
On page three they write:
Point (i.), in fact, turns out as an advantage for Earth-sized satellites of giant planets over terrestrial planets in terms of habitability, by the following reasoning: Application of tidal theories shows that the rotation of extrasolar planets in the IHZ around low-mass stars will be synchronized on timescales ≪1Gyr (Dole 1964; Goldreich 1966; Kasting et al. 1993). This means one hemisphere of the planet will permanently face the star, while the other hemisphere will freeze in eternal darkness. Such planets might still be habitable (Joshi et al. 1997), but extreme weather conditions would strongly constrain the extent of habitable regions on the planetary surface (Heath & Doyle 2004; Spiegel et al. 2008; Heng & Vogt 2011; Edson et al. 2011; Wordsworth et al. 2011). However, considering an Earth-mass exomoon around a Jupiter-like host planet, within a few million years at most the satellite should be tidally locked to the planet – rather than to the star (Porter & Grundy 2011). This configuration would not only prevent a primordial atmosphere from evaporating on the illuminated side or freezing out on the dark side (i.) but might also sustain its internal dynamo (iii.). The synchronized rotation periods of putative Earth-mass exomoons around giant planets could be in the same range as the orbital periods of the Galilean moons around Jupiter (1.7d$16.7d) and as Titan’s orbital period around Saturn ("16d) (NASA/JPL planetary satellite ephemerides)4. The longest possible length of a satellite’s day compatible with Hill stability has been shown to be about P"p/9, P"p being the planet’s orbital period about the star (Kipping 2009a). Since the satellite’s rotation period also depends on its orbital eccentricity around the planet and since the gravitational drag of further moons or a close host star could pump the satellite’s eccentricity (Cassidy et al. 2009; Porter & Grundy 2011), exomoons might rotate even faster than their orbital period.
So contemporary scientists agree with Dole that giant Jupiter sized worlds would tidally lock potentially habitable moons to them instead of to their stars. So far I don't see whether they also agree with Dole's calculations that satellites smaller than their planets could also tidally lock the planets to them.
I may write more on this later.
Added.
When Dole speculated about large satellites tidally locking their planets and so preventing the planets being tidally locked to their stars, our solar sysstm was the only one know. It includes terrestrial type planets close to the Sun, and giant planets far from the Sun where it is much too cold for liquid water using life.
And it was assumed that most solar systems would be like that, with the giant planets far from the stars where it was much too cold for our type of life. nobody in those days imagined that giant planets could get as to thier stars and as hot as Mars, or Earth, or Venus, or Mercury.
So Dole discusssed the possibilty that large satellites could prevent their planets from becoming tidally locked, and only briefly mentioned the possility of a habitable world being a giant satellite of a giant planet.
But when extrasolar exoplanets began to be discovered, the methods used made it much easier to detect very massive planets, and they made it much easier to detect planets that were very close to there stars. So guess what type of planet was very comommonly discovered in the early days? Giant planets very close to their stars, giant planets that were very, very hot.
So astonomers knew that if giant planets could orbit far from their stars and be far too cold for our type of life, as in our solar system, and if giant planets could also orbit very close their stars and be far too hot for our type of life, it was very propbable that some giant planets would orbit their stars at the right distances to have the right temperorures for life on their large moons.
So Heller and Barnes and the articles they cited naturally discussed the possiblity of worlds being prevented from being tidally locked to their stars by the tidal forces of giant planets that those world orbited, and which in turned orbited in the habitable zones of their stars.
And in recent years many more exoplanets have been discovered, and a wide varity of solar system types are known.
So today we know there are Earth size planets orbiting in the habitable zones of spectra l type G stars. Those Earth sized planets are far enough from their stars they don't need any giant moons to keep them from being tidally locked to their stars.
And today we know there are Earth size planets orbiting within the habitable zones of dim class K and class M type stars. These planets do need hypothetical large satellites (or possibly tidal interactions with other planets in the system) to keep them from being tidally locked to their stars.
And today we know there are giant size planets orbiting within the habitable zones of dim class K and class M type stars. Hypothetical moons of those planets would be tidally looked to their stars, except that the stronger tidal forces form the planets they orbit keep them tidally locked to their planets instead of to their stars.
What we don't know yet is whether any known exoplanets have any exomoons. There are a few candidates for exomoons, but as far as I no nobody has ever proved they have detected an exomoon yet. With more advanced techniques exommons may be discovered and confirmed.
Then we may know whether any Earth sized planets in the habitable zones of dim stars have large exomoons to potentially keep them from becoming tidally tocked to their stars. And we may know if any giant planets in the habitable zones of dim stars have large and potentially habitable exomoons orbiting them.
It has been claimed that an otherwise suitable planet would need to have a large moon to be habitable. That is a rather controversial opinion at present.
https://www.npr.org/2011/11/18/142512088/is-a-moon-necessary-for-a-planet-to-support-life
https://www.sciencedaily.com/releases/2022/02/220201144027.htm
https://www.rochester.edu/newscenter/moon-formations-habitable-exoplanets-509402/
So if a large moon is necessary for an Earthline planet to support life, an Earthlike planet in the habitable zone would need a large moon to be habitable anyway, and the even larger moon needed to prevent the planet from becoming tidally locked would not be that much of a difference. It would reduced the precentage of Earth sized planets in the habitable zones of red dwarf stars, but not much more than the requirement for smaller but still very large moons would already reduce it.
The answer to this question is positive. A substantial moon can synchronise the rotation rate of a planet.
The reason for this lies in the sharp frequency-dependence of the quality function of the planet (the Love number divited by the tidal quality function). While the peak magnitude of the solar torque may be comparable to or (for close-in planets) larger than the peak value of the lunar torque, it turns out that in this problem the interrelation between the peak magnitudes is irrelevant. Instead, the peak value of the lunar torque has to compete with a very small "tail" value of the solar torque. For a sufficiently massive moon, the lunar torque will win -- and will synchronise the planet's rotation.
For details, please see our recent work.
