Is it theoretically possible that planets exist with mountains so high that their peaks overtop the planets atmosphere? And which physical laws are relevant for this question? I'm just curious.
The question came while I imagined the atmosphere like a second ocean above our ocean of water. And I thought it would be nice if such a gaseous ocean could have islands as well.
This is a bit of a gray area, as an atmosphere doesn't have a clear boundary. That being said, Olympus Mons on Mars is so tall, the atmospheric pressure on top of it is only 12% the average pressure on the surface of Mars. That's near vacuum by terrestrial standards.
https://en.wikipedia.org/wiki/Olympus_Mons#Description
In general, for this to happen you need:
It's not a very likely combination, but it can happen, as seen on Mars.
Some planets have no atmosphere at all, so every bump, hill, and mountain would meet your requirement. There's no fundamental physics that has to do with the size of mountains and the thickness of the atmosphere. They are controlled by completely different processes.
There are obviously various definitions for the thickness of a planet's atmosphere. Atmospheric pressure and density drop roughly exponentially with height, $\rho = \rho 0.exp(-h/H)$. One can quote the characteristic scale height, $H$, over which this occurs. This is the height that the atmosphere would abruptly end if the density at the surface, $\rho 0$, were maintained all the way up.
For Earth, $H$ is about 8500 meters. Mount Everest is 8850 meters high, so by that definition it does poke out of the atmosphere, but barely.
For Mars, $H$ is about 11100 meters. Olympus Mons is about 21000 meters high, so by that definition it sticks out about twice the height of the atmosphere
For Venus, $H$ is about 15900 meters. Maxwell Montes is about 11000 meters high, so it sits well inside Venus' atmosphere
For Titan, $H$ is about 21000 meters. Mithrim Montes is only 3337 meters high so it stays right at the bottom of Titan's atmosphere.
Scale height is given approximately by $H = kT/mg$, where $k$ is Boltzmann's constant, $T$ is temperature, $g$ is local surface gravity, and $m$ is the mean mass of a molecule of the atmosphere.
The height of mountains is much more tricky. The work required to create a mountain will be proportional to $g$, so large planets with high surface gravity will tend to have smaller mountains. But the dynamic processes that create mountains will vary very much with the nature of the planet. Olympus Mons on Mars is a very large shield volcano that appears to have been active within the last few million years. The thin atmosphere probably results in little erosion and the relatively thick rigid crust and light gravity will likely mean relatively little hydrostatic subsidence over many eons.
As others have said, there is no sharp boundary for the atmosphere, but for Earth we have defined the Karman line, where a plane would have to be traveling at orbital velocity for its wings to provide enough lift to support itself. Even this value is not precise (air pressure can vary, for instance), but two widely-used values are 50 miles (about 80 km) and 100 km.
Interestingly, even though Mars has a very thin atmosphere, it fades away more slowly because of the planet's weaker gravity (it has a greater "scale height"). So the Karman line for Mars is thought to be about the same as for Earth, or three to four times the height of Olympus Mons. No orbiting satellite is going to collide with a Martian mountain.
You could certainly imagine a Martian atmosphere more tenuous than it actually is, so that the Karman line is only 20 km up, less than the 25 km height of Olympus Mons. In that sense, I agree with Brick that the answer can only be Yes. But the atmosphere in that case would be very thin indeed.
It would be interesting to know the thickest possible atmosphere for which a mountain can extend past the Karman line, but I don't have that answer.
I think the planet's size and it's geological activity will dictate the mountains height. Mars is smaller than Earth and no active core, So has high probability to have mountain like Olympus Mons.
