Given the horizontal coordinates of two objects in the sky, how does one go about calculating the angular separation between them?
You use Spherical Trigonometry
Given $A_1$ and $A_2$ are the respective azimuthal coordinates of the two objects, and $a_1$, $a_2$ their respective altitudes,
the angular seperation $\theta$ is given by
$$\cos \theta = \sin a_1 \sin a_2 + \cos a_1 \cos a_2 \cos (A_1-A_2)$$