In the same way that if you were to measure the height and speed of something thrown into the air, you would be able to work out how hard it had been thrown and how long ago. The assumption is made that the trajectory of the thrown object obeys the known laws of physics.
A further advantage one has in astronomy is that you are not limited to making a single measurement in the present day, you can observe distant things as they were in the past. This gives you multiple opportunities to measure the size and expansion rate of the universe to corroborate your answer and/or uncover holes in understanding of the physics (like dark energy for example). Nevertheless, to some extent, the age quoted does assume we have the physics right.
At the moment, the age estimate is heavily dependent on measurements of features that formed when the universe was about 400,000 years old (the cosmic microwave background), combined with observations of supernovae in galaxies at a range of distances (and times in the past), that tell us precisely how the trajectory of the universe has changed with time.
The accuracy of the answer (as opposed to the precision, which is very high), can be verified by independent tests like estimating the ages of the oldest stars, looking at how much structure there is in the universe, both now and in the past, etc.
Basically, everything ties up reasonably well at the moment (in fact there are small discrepancies in the expansion rate measured now, compared with predictions from the ancient cosmic microwave background), so the age you quote is an age for the "concordance model", that uses an agreed (though not entirely understood) set of physics.