I think we can work this our pretty well with a few approximations.
To start, let's say that the observable limit of the naked eye under completely dark skies is 6th magnitude in $V$. Let's also say that the human eye has an angular resolution of 1 arcminute, which will subtend $\pi \cdot 1^2 \approx 3$ square arcminutes on the sky.
Actual dark skies on Earth still have some measurable sky brightness - which is one reason we can't see arbitrarily faint stars - of about $21.8 \ \mathrm{mag} \over \text{arcsec}^2$ in V-band. See Figure 1 here.
The integrated brightness, $m$, of the sky background, $S$, will be:
$$m = S - 2.5 \cdot \log(A)$$
where $A$ is the angular area. Note that the integrated sky brightness will just scale along with $S$, the surface brightness of the sky, since $A$ remains constant.
So, if for $S = {21 \ \mathrm{mag} \over \text{arcsec}^2}$ the observable limit is 6th mag in $V$, the limit for arbitrary $S$ will be:
$$m_{\text{limit}} = 6 - [21 - S(z)]$$
where $S(z)$ is the sky brightness when the Sun is at some zenith angle $z$.
From that reference I linked to, $S = 21.8$ at about $z = 105^{\circ}$ (Fig 5), and it goes up pretty much linearly to $S = 10$ in $V$ at $z = 94^{\circ}$. So:
$$S(z) = 1.07 \cdot z - 90.5$$
Thus the limiting visible $V$ magnitude for a star, as function of the Sun's zenith angle ought to be roughly
$$m_{\text{limit}}(z) = 1.07 \cdot z - 105.5$$
But you wanted the altitude of the Sun below the horizon, not away from the zenith, so subtract $90^{\circ}$ from $z$ in the above:
$$m_{\text{limit}} = 1.07 \cdot \theta - 10$$
where $\theta$ is how far the Sun is below the horizon in degrees.
Approximately, at any rate.
