What is the mathematical formula used to do that? Is there a website/app that does the calculation for me?
It's a very general question, as a rule of thumb the higher an object is on the sky the light is going to be less dispersed as the atmosphere is thinner. So, the best point and moment to observe an object is when the object is in zenith (directly over your head). Usually it is not possible to travel around the globe to make your observations, so we are going to see better the objects that happen to be near zenith, or they that are crossing the meridian line (with circumpolar objects, it's better wait until the object the meridian line nearest to the zenith)
I don't have here my books so I can't give you a proper formula, but if you want, I can try to give you that formulae.