$H_0^{-1}$ is only a rough estimate for the age of the universe and you have correctly identified the reasons why not.
A correct age estimation relies on knowing $H_0$ and the densities of matter and dark energy, so that the past expansion history of the universe can be correctly modelled. Even this relies on an assumption about how dark energy behaves.
A more interesting question is why a value of $H_0 \simeq 70$ km s$^{-1}$/ Mpc gives an $H_{0}^{-1}$ of 14 billion years, which is within a few per cent of the current best estimate for the age of the universe of 13.8 billion years. In the $\Lambda$CDM cosmology model, the reason for this cosmic coincidence is that the universal expansion underwent a period of decelaraton up until about 4 billion years ago, when it started to accelerate again (the red curve on the plot below).
As a result, a tangent to the red curve, showing the size of the universe versus time, at the present epoch almost goes to ${\rm size}=0$ about 14 billion years ago. If we were to go backwards or forwards in time by 5 billion years, the agreement between $H_0^{-1}$ and the age of the universe would not be so good. At earlier times $H_0^{-1}$ would have overestimated the age of the universe, whereas at later times $H_0^{-1}$ will underestimate the age of the universe.
