The problem is not with the spherical triangle. A triangle can always be formed between three points on a sphere that are not on a great circle.

The problem is the triangle cannot be drawn with an altitude a=0 if:

• the object never rises. (Point X is always below the horizon.)
• the object never sets. (Point X is between the horizon and the celestial pole P, thus the three points X, P, and the zenith Z are on a great circle.)

The hour angle H in the rising and setting formula is the number of hours from the time the object rises to the time when it is on the meridian. (The meridian is the circle going through the celestial pole P and the zenith Z in the diagram.) Or another way to say it, 2*H is the time from rising to setting. When the object never rises, there is no value of H that satisfies the equation. If the object never sets, there is no value of H that satisfies the equation.

The problem is not with the spherical triangle. A triangle can always be formed between three points on a sphere as long as the three points are not on a great circle. (The three points are the celestial pole P, the zenith Z, and the object X in the figure.)

The problem is the triangle cannot be drawn with an altitude of 0 for X if:

• the object X never rises.
• the object X never sets.

Since the spherical triangle cannot be created in the two cases, the value of the hour angle H cannot be found. 

Or another way to say it, the time 2*H is the time from rising to setting. When the object never rises, there is no value of H that satisfies the rising and setting formula. If the object never sets, there is no value of H that satisfies the rising and setting formula. In both cases, the right side of the equation is greater than 1 or less than -1, but the value of cos H is limited to ranges between -1 and 1.