Breaking the phrase down:
- Dwarf star - a term I will never understand - is used to describe relatively small, dim stars. Unfortunately, this encompasses most main-sequence stars, which are indeed dwarfs compared to large giants and supergiants.
Ultra-cool, as called2voyage already discussed, means that the star has an
effective temperatures of less than 2,700 kelvin.
This is about half of the Sun's surface temperature. Spectroscopically, it means that these stars are of class M7-L8, i.e. really, really cool stars that just barely reach the threshold for nuclear fusion. Actually, some L dwarfs will never fuse hydrogen as normal stars do, and will become brown dwarfs. Therefore, "ultra-cool dwarf" doesn't necessarily refer to just main sequence stars.
Other characteristics in some cases (from Cruz et al.):
- Lithium in some L-type objects, indicative of low temperatures, low masses, and young ages in those particular objects.
- Very low surface gravities in late M-type dwarfs (though likely not as much in an M7 dwarf like TRAPPIST-1).
- Very low metallicity in certain other L-type objects.
TRAPPIST-1 actually appears to be a rather massive ultra-cool dwarf, then, in comparison to some of these other objects.
Here's an annotated Hertzsprung-Russell diagram, with a box for ultra-cool dwarfs and a circle for where TRAPPIST-1 is, approximately:
Image from Wikipedia courtesy of user Saibo, under the Creative Commons Attribution-Share Alike 2.5 Generic license.