# How precise are Saemundsson/Bennett refraction formulas?

I was trying to reproduce the predictions of the Bennett formula* by physically-based calculations with a model based on a real-life refractivity profile. My calculations based on Table V from ref. 1 compare as follows with the Bennett formula:

As you can see, there's some noticeable discrepancy, that's larger than between e.g. inverted-Saemundsson formula and directly-used Bennett formula (which would blend on a plot).

I've tried to come up with a refractivity profile that would reproduce the Bennett formula much better. I've indeed been able to make a very simple exponential profile that reproduces the values much better:

But the refractivity fitted is very different from that in ref. 1:

So I think that Bennett formula must actually be quite imprecise, and I shouldn't try too hard to fit the model to it. But how much is it actually imprecise compared to measured astronomical refraction? Are there any freely accessible measured values of refraction itself rather than refractivity profiles, that could be used to estimate the precision of Bennett formula (and of my calculations)?

### References

1. D. Vasylyev, W. Vogel, "Satellite-mediated quantum atmospheric links", Phys. Rev. A 99, 053830 (2019)

*Per Wikipedia:

$$R = \cot \left( h_a + \frac{7.31}{h_a + 4.4} \right)$$

where $$R$$ is the refraction in arcminutes and $$h_a$$ is the apparent altitude of the astronomical body in degrees.

• Cool question! I wonder why the reference broke the atmosphere up into only 10 discrete layers; it seems the model could easily be implemented via numerical integration with much smaller step sizes; the U.S. Standard Atmosphere, 1976 model they use has high granularity tabular data. Since giant step size approximations to monotonic behavior can lead to a systematic over or under prediction, it might be the cause of the difference in your $(n-1) \times 10^8$ plot.
– uhoh
Jun 23, 2021 at 2:13
• First three points at 0, 5 and 7 km are hugely spaced! Note they agree at 0 and monotonically deviate thereafter. Perhaps their values are effective ones for the contribution throughout the thick layer whereas the other in your plot is the index value at the layer's nominal height only?
– uhoh
Jun 23, 2021 at 2:15
• @uhoh hmm, re-reading my ref. 1, I've indeed misused their table. Right above it they describe that $n_i(h)$ is linearly-changing in the $i$th segment, while I simply interpolated between the points on the log scale. So that table seems to be a poor reference. Too bad the US Standard Atmosphere doesn't explicitly specify refractivity, one has to derive it from the constituents... Jun 23, 2021 at 9:04
• – uhoh
Jun 24, 2021 at 3:41

First of all, it must be noted that the solution to the problem of finding a refractivity profile from a ground-level dependence of refraction on elevation is not unique — at least if we just aim to get a good approximation. A pair of drastically different profiles can agree within the same error bounds on the astronomical refraction.

Now, ref. 1 in the OP gives quite an imprecise profile that approximates each (multi-km) altitude segment with a linear function. The OP used the data given in ref. 1, interpolating between the points on a logarithmic scale, which may have broken the approximation completely. So, the computation of astronomical refraction there is unreliable.

We can do better, namely, take profiles from ref. A that give more detailed (and more complete — up to 120 km) profiles of atmospheric densities and volume percentages of various constituents. Using the simplistic approach, i.e. assuming that the refractivity of air depends only on number density at each altitude (and independent of actual constitution) will give us, for $$\lambda=555\,\mathrm{nm}$$ (with $$n$$ taken from here), the following refractivity profile for subarctic summer conditions in ref. A:

We can see that the values in ref. 1 from the OP are not that far from our new profile, and both are very far from the OP's fit. At the same time, the difference from the Bennett's formula (adjusted to the same ground temperature of $$287.2\,\mathrm{K}$$ from its default $$283\,\mathrm{K}$$) is already quite small:

Peak difference from the Bennett formula is about $$0.0072°$$, which is comparable to the peak difference of the OP's fit that's about $$0.0023°$$, and is much less than the difference of $$0.042°$$ in the OP's physically-based calculations. A simple hack of using the temperature less by $$2.9°$$ than actual AFGL data when generating the refractivity profile yields an error of $$0.0015°$$ that's even smaller than that of the OP's fit. This is a cheat of course, but this just shows the non-uniqueness of the solution. The correct way would be to take refractivity of each constituent and compute the appropriate value for each altitude separately.

As for the Bennett formula's precision, Wikipedia cites Bennett's own paper, ref. B, saying that its error is within $$0.07'=0.0012°$$ from ray tracing calculations with the algorithm from ref. C. So we can consider this to be the answer to the OP.

### References

B. Bennett, G. (1982). The Calculation of Astronomical Refraction in Marine Navigation. Journal of Navigation, 35(2), 255-259. doi:10.1017/S0373463300022037

C. B. Garfinkel. Astronomical refraction in a polytropic atmosphere. Astronomical Journal, Vol. 72, p. 235-254 (1967). doi:10.1086/110225