I remember seeing in a video an instrument that was used to measure a star's radius with nothing more than the light it emitted. The device worked by opening or closing its mechanism in increments, while watching a display. The display started out showing some shape, maybe some lines or a circle, with blurriness. The closest you could get to making the shape blurry-less would determine the instrument's approximation of the star's radius. My question is, what is the name of this instrument?
I am not sure what you are referring to; possibly it is intensity interferometry using the Hanbury Brown & Twiss method, but even this requires two telescopes. This kind of interferometry was used to estimate angular diameters for some tens of stars but is not generally used these days for this purpose. Instead, one constructs interferometers from separated telescopes using delay lines to compensate for path-length differences and maintain coherence between the different paths that the light takes. This then uses amplitude interferometry to reconstruct the angular diameter of a star from the interference pattern seen in different telescope baseline configurations.
The radii (or to be more exact, the angular size) of many nearby stars and also of many giant stars at rather larger distances, have been measured using interferometric techniques. An accurate distance is then required to estimate an absolute radius.
There are also direct radius measurements for stars that are occulted by the Moon and other solar system bodies - the time-dependent diffraction pattern can yield an angular size for the object.
Absolute radii are also provided by eclipsing binary systems (independent of whether they have a known distance).
Measuring a star's radius directly (rather than inferring it through empirical relations to other observables) is extremely difficult. As barrycenter points out, it can't be done for most stars because they're too far away. However, we have been successful in measuring the radius of a small selection of very close stars.
This process is primarily done using infrared-wavelength interferometry. For example, this method has been used to directly measure the radius of Betelgeuse, using the Infrared Spatial Interferometer (ISI). I have no source for this, but I believe the USNO has performed a similar measurement for about 20 stars (I recall this from a seminar I saw years ago).
Using interferometry is the only way I know that a stellar radii has directly been observed. I really don't have any guess as to what you saw.