Short Answer:
In part Two of the long answer below, it is stated that a planetary mass object needs to have a mass of at least 0.1, 0.12, 0.23, or 0.25 Earth mass to be habitable. Worlds with those masses and with a radius of 0.58 Earth radius should have densities of 2.827, 3.392, 6.502, or 7.067 grams per cubic centimeter. Such densities are possible, though it would be unusual for such small mass worlds to have such high densities.
If such a world is made more dense, it could have such a mass with a radius a bit smaller than 0.58 Earth radius, though a small world with such a high density would be even rarer and more unlikely.
Taking planetary density to a theoretical extreme, if a planetary mass object was made out of osmium, rough calculations indicate it could have those minimum masses with radii of 0.29 to 0.3945 of the radius of the Earth.
And of course several solar system bodies with no atmosphere and surfaces of solid ice are believed to have liquid water in subsurface oceans, and thus possibly be habitable for alien life in their subsurface oceans. The smallest of them of them is Enceladus, a moon of Saturn, with a radius of only about 252.1 kilometers, or 0.039 the radius of the Earth.
Long Answer:
When you say "habitable" do you mean "habitable for carbon based liquid water using organisms in general" or "habitable for human beings, other intelligent beings, and multicellular land animals that need an oxygen rich atmosphere in particular"?
Part One of Two: Habitable for multicellular land animals that need an oxygen rich atmosphere.
As far as I know, the only scientific study of what planets would be habitable for multicellular land animals that need an oxygen rich atmosphere in general, and for humans in particular, is Habitable Planets for Man Stephen H. Dole, 1964, 2007.
https://www.rand.org/content/dam/rand/pubs/commercial_books/2007/RAND_CB179-1.pdf[1]
Dole believed that there are formulas for calculating the relationships between
the mass of a terrestrial type planet and its radius, volume, density, surface gravity, and escape velocity. Thus the mass of a terrestrial type planet would determine its radius, volume, density, surface gravity, and escape velocity. And that is largely true for the few terrestrial type worlds in our Solar System, which formed from the same protoplanetary disc with similar percentages of different elements with different densities.
But even in our Solar System Mercury is much more dense than such a small planet should be. It is believed that billions of years ago Mercury collided with an even smaller planet, and most of the lighter elements in Mercury's crust were lost in the impact, leaving only the denser elements behind.
And in the outer Solar System, where the protoplanetary disc included a higher percentage of the lighter elements, many of the large, planetary mass, moons and other objects have, are made of mixtures of ices of various liquids with rock, and so are less dense than the terrestrial planets.
Astronomers have discovered thousands of exoplanets orbiting other stars, and have measured or calculated the masses, radii, densities, surfaces gravities, and escape velocities of some of them. Planets formed out of the protoplanetary discs of other stars with other percentages of various elements are likely to have somewhat different densities than terrestrial planets in our solar system.
A few large exoplanets are believed to have very high densities, and it is theorized that they are the remaining dense rocky cores of giant planets that lost most of their light elements in various cataclysms.
So it is possible for a science fiction writer to justify a terrestrial type planet with a density somewhat different than what would be calculated based on the terrestrial type planets in our Solar System. But not too much different.
On page 53 Dole sets a rough maximum possible mass for a planet habitable for humans.
"Now it will be recalled that, to be considered habitable, a planet must have a surface gravity of less than 1.5 g. From figure 9, it may be seen that this corresponds to a planet having a mass of 2.35 Earth masses, a radius of 1.25 Earth radii, and an escape velocity of 15.3 kilometers per second.
Of course an exoplanet which formed from a protoplanetary disc around a different star, might be made of matter with a higher or lower average density, and so might have a slightly greater mass and radius than Dole calculated, while still having a surface gravity of 1.5 g or less.
And of course lifeforms evolving on an alien exoplanet could be capable of surviving in a much higher surface gravity than 1.5 g.
So what did Dole calculate for the minimum mass of a planet with an oxygen rich atmosphere habitable for humans, and would the minimum mass of a planet with liquid water on its surface be any higher or lower?
On page 54 Dole says:
However, if we take as a rough approximation that maximum exosphere temperatures as low as 1000 degree K are not incompatible with the required surface conditions of a habitable planet, then the escape velocity of the smallest planet capable of retaining atomic oxygen may be as low as 6.5 kilometers per second (5 X 1.25). Going back to figure 9, this may be seen to correspond to a planet having a mass of 0.195 Earth mass, a radius of 0.63 Earth radius, and a surface gravity of 0.49 g.
Dole believed that a planet that small could retain an oxygen rich atmosphere, but didn't think that it produce an oxygen rich atmosphere in the first place. Thus a planet with 0.195 of Earth Mass would probably not form a natural oxygen rich atmosphere but could be terraformed and given an oxygen rich atmosphere by an advanced civilization.
Dole used 2 different possible rules to calculate 2 different possible minimum masses for a planet with an oxygen rich atmosphere. Those calculations produced a minimum mass of 0.25 and 0.59 Earth mass respectively. So on pages 56 and 57 Dole decided that the true value would be intermediate and chose a value of about 0.4 Earth mass.
This corresponds to a planet having a radius of 0.78 Earth radius and a surface gravity of 0.68 g.
So a planet with the mass of 0.4 Earth mass would have a radius of 0.78 Earth, which is 1.34 times a radius of 0.58 Earth radius, and even a planet with a mass of 0.195 Earth would have a radius of 0.63 Earth radius, which is about 1.08 times a radius of 0.58 Earth radius.
And I guess that 0.63 Earth radius is close enough to 0.58 Earth radius, that a planet with a lesser density and more light elements could possibly produce an oxygen rich atmosphere and replace it as fast as it loses oxygen from the exosphere. Alternatively, a planet made of denser materials might have a high enough escape velocity and produce enough oxygen to retain an oxygen rich atmosphere. Possibly, that might stretch the minimum radius for habitability down to 0.58 Earth radius.
Part Two of Two: Habitable for carbon based liquid water using lifeforms.
Such a planet would not need to have an oxygen rich atmosphere, just a dense enough atmosphere to have sufficient pressure for water to be liquid at most temperatures on the planet's surface. The lower the atmospheric pressure of a planet is the lower the boiling temperature of water is, until with a low enough atmospheric pressure ice will turn directly into water vapor without ever being a liquid.
"Exomoon Habitability Constrained by Illumination and Tidal Heating", Rene Heller and Roy Barnes, Astrobiology, Volume 13, number 1, 2013, is a comparatively recent discussion of the possible habitability of worlds in other solar systems, in this case as yet undiscovered exomoons orbiting exoplanets in other solar systems. Heller and Barnes do not mention any reason to suppose that exomoons could be habitable if they were outside the mass range for an exoplanet to be habitable. Therefore they summarize scientific opinions about the mass range for potentially habitable worlds.
https://faculty.washington.edu/rkb9/publications/hb13.pdf[2]
In section 2. Habitability of Exomoons, the last paragraph before section 2.1 2.1. Formation of Massive Exomoons, on page 20, discusses the mass range for habitable worlds, including exoplanets and exomoons.
A minimum mass of an exomoon is required to drive a
magnetic shield on a billion-year timescale (MsT0.1M4;
Tachinami et al., 2011); to sustain a substantial, long-lived
atmosphere (MsT0.12M4; Williams et al., 1997; Kaltenegger,
2000); and to drive tectonic activity (MsT0.23M4; Williams
et al., 1997), which is necessary to maintain plate tectonics
and to support the carbon-silicate cycle. Weak internal dynamos have been detected in Mercury and Ganymede
(Gurnett et al., 1996; Kivelson et al., 1996), suggesting that
satellite masses > 0.25M4 will be adequate for considerations
of exomoon habitability. This lower limit, however, is not a
fixed number. Further sources of energy—such as radiogenic
and tidal heating, and the effect of a moon’s composition and
structure—can alter the limit in either direction. An upper mass limit is given by the fact that increasing mass leads to high pressures in the planet’s interior, which will increase the mantle viscosity and depress heat transfer throughout the mantle as well as in the core. Above a critical mass, the dynamo is strongly suppressed and becomes too weak to generate a magnetic field or sustain plate tectonics. This maximum mass can be placed around 2M4 (Gaidos et al., 2010; Noack and Breuer, 2011; Stamenkovic´ et al., 2011).
Their source for a minimum mass of 0.1 Earth Mass for a magnetic shield is
Tachinami, C., Senshu, H., and Ida, S. (2011) Thermal evolution
and lifetime of intrinsic magnetic fields of super-Earths in
habitable zones. Astrophys J 726, doi:10.1088/0004-637X/726/
2/70.
Their source for a minimum mass of 0.12 Earth mass for a substantial, long lived atmosphere is
Williams, D.M., Kasting, J.F., and Wade, R.A. (1997) Habitable
moons around extrasolar giant planets. Nature 385:234–236.
Their source for a minimum mass of 0.23 Earth mass for plate tectonics is
Williams, D.M., Kasting, J.F., and Wade, R.A. (1997) Habitable
moons around extrasolar giant planets. Nature 385:234–236.
Their sources for the weak magnetic fields of Mercury and Ganymede are:
Gurnett, D.A., Kurth, W.S., Roux, A., Bolton, S.J., and Kennel,
C.F. (1996) Evidence for a magnetosphere at Ganymede from
plasma-wave observations by the Galileo spacecraft. Nature
384:535–537.
Kivelson, M.G., Khurana, K.K., Russell, C.T., Walker, R.J.,
Warnecke, J., Coroniti, F.V., Polanskey, C., Southwood, D.J.,
and Schubert, G. (1996) Discovery of Ganymede’s magnetic
field by the Galileo spacecraft. Nature 384:537–541.
Their source for the importance of plate tectonics for habitability is:
Williams D.M. Kasting J.F. Wade R.A. Habitable moons around extrasolar giant planets. Nature. 1997;385:234–236. [PubMed] [Google Scholar]
Their sources for that maximum mass limit of 2 times the mass of Earth are:
Gaidos, E., Conrad, C.P., Manga, M., and Hernlund, J. (2010)
Thermodynamics limits on magnetodynamos in rocky exoplanets. Astrophys J 718:596–609.
Noack, L. and Breuer, D. (2011) Plate tectonics on Earth-like
planets [EPSC-DPS2011-890]. In EPSC-DPS Joint Meeting 2011,
European Planetary Science Congress and Division for Planetary Sciences of the American Astronomical Society. Available
online at http://meetings.copernicus.org/epsc-dps2011.
Stamenkovic´, V., Breuer, D., and Spohn, T. (2011) Thermal and
transport properties of mantle rock at high pressure: applications to super-Earths. Icarus 216:572–596.
The density of the planet Earth is 5.513 grams per cubic centimeter. A sphere with a radius of 0.58 Earth radius would have a volume of 0.195 of the volume of Earth.
So putting a mass of 0.1 Earth mass within a volume of 0.195 Earth volume would result in a density of 0.512 of Earth's density or 2.827 grams per cubic centimeter.
So putting a mass of 0.12 Earth mass within a volume of 0.195 Earth volume would result in a density of 0.615 of Earth's density or 3.392 grams per cubic centimeter.
So putting a mass of 0.23 Earth mass within a volume of 0.195 Earth volume would result in a density of 1.179 of Earth's density or 6.502 grams per cubic centimeter.
So putting a mass of 0.25 Earth mass within a volume of 0.195 Earth volume would result in a density of 1.282 of Earth's density or 7.067 grams per cubic centimeter.
In a comment, Xi-k says that since the relatively common element iron, a major component of the cores of most planets, has a normal density of 7.874 grams per cubic centimeter, higher than the figure of 7.067 grams per cubic centimeter quoted above, and the iron in the cores of planetary bodies is compressed to greater densities than 7.874 grams per cubic centimeter, a mostly iron planet could be habitable with a small radius. Thus a planet 80 to 90 percent iron with 10 to 20 percent silicate crust could have an average density of about 7.067 grams per cubic centimeters.
This seems like a plausible way to get the radius of a habitable planet down to about 0.58 Earth radius. But an expert on planetary development could probably give a better estimate about the probability of planets being that small and dense.
At at the extreme of physical possibility, and very, very, very, very improbable, is a small planetary object made mostly of osmium, a very, very rare element. Such a planet might have to be created by a civilization collecting osmium from several solar systems and building a planet from it.
Osmium is the densest naturally occurring element, with an experimentally measured (using x-ray crystallography) density of 22.59 g/cm3.
https://en.wikipedia.org/wiki/Osmium[3]
So a sphere of solid osmium, with a thin layer of lighter materials necessary for life, could possibly exist - without discussing the probability of such a sphere forming.
Ignoring the compression and increased density of a planetary sized osmium sphere, and assuming that the hypothetical planet has an average density 4.0915875 that of Earth, it could have mass of 0.1 to 0.25 Mass earth with a radius of about 0.29 to 0.3945 Earth.
Added NOv. 17, 2021:
In my answer to this question:
https://worldbuilding.stackexchange.com/questions/217618/how-would-an-osmium-core-affect-life-on-a-planet/217651#217651
I designed planets with iridium cores, not osmium cores. Iridium, as PM 2Ring pointed out in his comment, is much less dangerous for life.
If a planet with an iridium core has to have a surface gravity low enough for humans, and also an escape velocity hight enough to retain an oxygen rich atmosphere, the planet may have to have a greater radius than a planet habitable by microbes or sea life, which not be bothered by high surface gravity.
