The present expansion of the universe follows Hubble's law
$$\frac{da}{dt} = H a\ , $$
where $H_0$ is the Hubble parameter and $a$ represents a scale factor (perhaps the distance between two galaxies).
The age of the universe will only be $H^{-1}$ at any epoch if the expansion rate itself, $da/dt$ is constant. This would imply a Hubble parameter varying as $H=H_0a^{-1}$, where $H_0$ is the present Hubble parameter at a scale factor of $a=1$, and would be the case for an unrealistic universe with just curvature and no matter, radiation or dark energy, since the evolution of $H$ is governed by
$$H^2 = H_0^2 \left( \frac{\Omega_r}{a^4} + \frac{\Omega_M}{a^3} + \frac{\Omega_k}{a^2} + \Omega_{\Lambda}\right)\ .$$
Here, $H_0$ is the Hubble parameter now, $a(t)$ is the scale factor of the universe, $\Omega_r$ is the current (i.e. $a=1$) ratio of the radiation density to the critical density and $\Omega_M$, $\Omega_k$ and $\Omega_{\Lambda}$ are the equivalent densities for the matter (baryonic and dark), curvature and (constant) vacuum energy densities.
In other, more realistic universes then $H$ changes in different and more complex ways and the age of the universe is some multiple of $H_0^{-1}$, and not exactly $H^{-1}$ at other times. That is because the Hubble parameter changes and it changes in a way that is not just proportional to $a^{-1}$.
Your idea of a universe with a Hubble parameter that does not change with time would be a universe that has always been dominated by dark energy. In this case, the expansion is exponential; $H = H_0 \Omega_\Lambda^{1/2}$ and rolling the clock back, the separation between two objects would never be zero, but would obey
$$ a = \exp[H_0 \Omega_\Lambda^{1/2}(t-t_0)] \ , $$
and the age of the universe would be infinite. This is also unrealistic, since whilst dark energy has a fixed density, matter and radiation densities were much bigger when the universe was smaller.
The age of the universe actually is quite close to $H_0^{-1}$ because of the "cosmic coincidence" that we live in a universe where the matter and dark energy densities are similar. The decelerating effects of the matter are cancelled out, and now exceeded, by the dark energy, resulting in an $a$ versus $t$ relationship which has a tangent ($da/dt= H_0$) at the current epoch ($a=1$) that can be extrapolated back to $a=0$ at roughly the correct time of the big bang. See Why can we trust Hubble Time if the rate of expansion is not constant? .
