The density of matter depends not only on its composition, but also on temperature and pressure. It's not meaningful to say that substance A is denser than substance B without specifying the conditions under which the comparison is being made.

For a simple everyday example, at room temperature (and pressure) water is significantly denser than air. But heat them both above 100 °C, and the water evaporates and actually becomes considerably less dense than air, even at the same temperature and pressure.

(By the ideal gas law, the density of different gases at a given temperature and pressure is approximately proportional to their average molecular mass. The molecular mass of water is only about half that of diatomic oxygen and nitrogen, which are the main components of air on Earth, and thus water vapor is only about half as dense as air at the same temperature and pressure.)

The surface temperature of Mercury is less than 1000 °C (and the interior temperature should not be much greater), and it mostly consists of metals and silicate minerals (i.e. rock) that are solid or liquid at those temperatures. The Sun's temperature, meanwhile, is over 5000 °C at the surface (photosphere), and a lot hotter deeper inside. If you could heat Mercury up to the same temperature as the Sun, most of the rocks and metals it consists of would evaporate, and would become a lot less dense. So a lot of the density difference simply comes down to the fact that Mercury is a lot cooler than the Sun, and thus able to stay solid.

Another reason why the Sun is less dense than Mercury is that the Sun contains a lot of lightweight hydrogen gas (which has both a very low molecular weight and a very low evaporation point), while Mercury has almost no hydrogen at all. The main reason for this is that the Sun's heat and the solar wind have effectively blown away any hydrogen and other volatile low-density substances that Mercury might once have had (or that might've existed in its general area while the solar system was forming).

The Sun itself can retain hydrogen due to its enormous gravity (but even so, it loses about one billion kilograms of it per second; that's basically what the solar wind I mentioned above mostly is). Mercury, however, is much smaller, and thus its gravity isn't strong enough to hold onto its own hydrogen so close to the Sun.

(Basically the same thing happened to Venus, Earth and Mars, which is why these inner planets didn't turn into huge balls of hydrogen gas like Jupiter and Saturn did. However, Earth and Venus were both big enough, and located far enough from the Sun, that they could hang on to other slightly less volatile substances like water and air. Mars is located even further from the Sun, but is also a lot smaller than Earth, which is the main reason why it today has only a very thin atmosphere of carbon dioxide, and very little if any water.)

The density of matter depends not only on its composition, but also on temperature and pressure. It's not meaningful to say that substance A is denser than substance B without specifying the conditions under which the comparison is being made.

For a simple everyday example, at room temperature (and pressure) water is significantly denser than air. But heat them both above 100 °C, and the water evaporates and actually becomes considerably less dense than air, even at the same temperature and pressure.

(By the ideal gas law, the density of different gases at a given temperature and pressure is approximately proportional to their average molecular mass. The molecular mass of water is only about half that of diatomic oxygen and nitrogen, which are the main components of air on Earth, and thus water vapor is only about half as dense as air at the same temperature and pressure.)

The surface temperature of Mercury is less than 1000 °C (and the interior temperature should not be much greater), and it mostly consists of metals and silicate minerals (i.e. rock) that are solid or liquid at those temperatures. The Sun's temperature, meanwhile, is over 5000 °C at the surface (photosphere), and a lot hotter deeper inside. If you could heat Mercury up to the same temperature as the Sun, most of the rocks and metals it consists of would evaporate, and would become a lot less dense. So a lot of the density difference simply comes down to the fact that Mercury is a lot cooler than the Sun, and thus able to stay solid.

Another reason why the Sun is less dense than Mercury is that the Sun contains a lot of lightweight hydrogen gas (which has both a very low molecular weight and a very low evaporation point), while Mercury has almost no hydrogen at all. The main reason for this is that the Sun's heat and the solar wind have effectively blown away any hydrogen and other volatile low-density substances that Mercury might once have had (or that might've existed in its general area while the solar system was forming).

The Sun itself can retain hydrogen due to its enormous gravity (but even so, it loses about one billion kilograms of it per second; that's basically what the solar wind I mentioned above mostly is). Mercury, however, is much smaller, and thus its gravity isn't strong enough to hold onto its own hydrogen so close to the Sun.

(Basically the same thing happened to Venus, Earth and Mars, which is why these inner planets didn't turn into huge balls of hydrogen gas like Jupiter and Saturn did. However, Earth and Venus were both big enough, and located far enough from the Sun, that they could hang on to other slightly less volatile substances like water and air. Mars is located even further from the Sun, but is also a lot smaller than Earth, which is the main reason why it today has only a very thin atmosphere of carbon dioxide, and very little if any water.)