Before protostars become "stars" and begin nuclear fusion, they are just balls of gas. Proto- brown dwarfs are also just balls of gas, but their cores never become hot enough for significant hydrogen ignition.

Roughly speaking, for a ball of gas, its contraction timescale is given by the amount of thermal energy it has divided by the rate at which it loses energy from its surface.

The amount of thermal energy depends on mass multiplied by the internal temperature. The luminosity at the surface also depends indirectly on mass, but with a steeper dependence than the thermal energy. This means that the contraction timescale is always longer for lower mass objects.

Before protostars become "stars" and begin nuclear fusion, they are just balls of gas. Proto- brown dwarfs are also just balls of gas, but their cores never become hot enough for significant hydrogen ignition.

Roughly speaking, for a ball of gas, its contraction timescale is given by the amount of gravitational potential energy it has divided by the rate at which it loses energy from its surface.

The amount of potential energy energy depends on mass squared. The luminosity at the surface also depends indirectly on mass, but with a steeper dependence than the potential energy. This means that the contraction timescale is always longer for lower mass objects - roughly as $M^{-1}$ for low-mass objects.