Okay, so I've seen different notations for the stars in β Capricorni. Some sources state that the stars are Aa, Ab1 and Ab2, Ba, and Bb, while others say that they are Aa, Aba and Abb, Ba, and Bb. So, what is the proper way to name stars in a multiple star system, and are there any exceptions to this convention?
The purpose of the notation is to indicate how the system is physically constructed, to indicate which stars orbit which - especially useful in hirachically organized stellar systems. So both ways are correct as their meaning is understood.
The notation I encounter most often is the first one you indicate: indicating with capital letters the systems which orbit the common pericenter of the overall system. With non-capital letters the components of the upper level are distinguished. And keeping it more readable, numbers are used for the 3rd level if further distinction is needed for a 3rd level.
However this nomenclature is not unique - and there is so far no general consensus. See the Washington Multiplicity Catalog for a take on getting some order at a given time and the IAU working group on stellar names. For a given system the designation might be more traditional and depend on order of discovery and naming by discoverer than following a fixed scheme. See also the discussion of future nomenclature on stellar system components in the corresponding wiki article.
I suppose that the International Astronomical Union has rules for naming stars in multiple systems.
I believe the usual rule is to describe the brighter star as A and the dimmer star as B, which works well as long as neighter star is very variable.
Thus the brighter star in Alpha Centauri is Alpha Centauri A and the less bright star is Alpha Centauri B. Sometime later a very dim red dwarf was discovered close to Alpha Centauri A & B, and became known as Alpha Centauri C, or Proxima Centauri since it is the closest star to our solar system. I think that recent research has shown that Alpha Centauri C is actually gravitationally bound to A and B and part of their star system.
I also know that when exoplanets are found orbiting stars the exoplanets are usually given lower case Latin letters, starting with b for the first exoplanet discovered. And there is an exception when several exoplanets are discovered orbiting the same star at the same time. In that case they are designated b, c, d, etc. in order of increasing distance from the star. And if more exoplanets are later discovered in the system, planet e could orbit between planets b and c, for example.
That is different from the system in science fiction stories where it is normal to designate exoplanets with Roman numerals in order of increasing distance from their star, so that in our solar system Mercury is sometimes called Sol I, Venus is sometimes called Sol II, Earth is sometimes called Sol III, and so on.
Castor, or Alpha Geminorum, is a sextuple star system, with visible three stars which are spectroscopic binaries. That means that telescopes show three Castor stars as dots of light, but the spectra of each star shows that it is actually a close binary star with two stars revolving around each other and thus changing their doppler shifts relative to Earth.
Castor is a multiple star system made up of six individual stars; there are three visual components, all which are spectroscopic binaries. Appearing to the naked eye as a single star, Castor was first recorded as a double star in 1718 by James Pound, but it may have been resolved into at least two sources of light by Cassini as early as 1678. The separation between Castor A and Castor B has increased from about 2″ (2 arcseconds of angular measurement) in 1970 to about 6″ in 2017. These two binary pairs have magnitudes of 1.9 and 3.0.
So sometime in or after 1718 the two brightest strs in Castor were labeled Castor A and Castor B. Sometime later the third visible star was discovered and named castor C, or Alpha Geminorum C.
When the three stars A, B, & C were discovered to be spectroscopic binaries, the two stars in each pair were give lower case letters a & b, so the six stars are now named Castor Aa and Ab, Castor Ba and Bb, and Castor Ca and Cb. That doesn't conflict with the system of naming exoplanets as long as any expolanets in the Castor system orbit Castor Aa, Castor Ba, or Castor Ca. but if exoplanets are found orbiting Castor Ab, Castor Bb, or Castor Cb, I'm not certain how they would be labeled.
Fortunately the orbital periods of Castor Aa & Ab, and Castor Ba & Bb, are only a few days long, and the orbital period of Castor Ca & Cb is less than a day. Thus most exoplanets found in the Castor system would probably orbit around both the stars in a pair. And I am not certain how such circumbinary exoplanets would be labeled. Probaly as Castor A b, c, d, etc. Caster C c, d, e, etc. and so on.
So I think that with the present system there is a possibility of ambiguous labeling, and people being uncertain whether Castor Ab, for example, is a star or an exoplanet.
I note that the original question mentions stars labeled with an Arabic number as part of a multiple system. And I never heard of that before.
And no doubt an astronomer who studies multiple stars can give a much better explanation of the naming conventions.