The "lithium test" for a brown dwarf involves measuring two things - the lithium content and the spectral type (a proxy for surface temperature) or luminosity. But you may also need to know (or assume) something about the age of the object. The method is useful because the tracks of low-mass stars and brown dwarfs run together very closely in the usual luminosity versus Teff plane of the Hertzsprung-Russell diagram. In addition to the considerable uncertainty in the placement of these tracks, one really needs to know the age of an object in order for its luminosity or Teff to lead to even a crude mass estimate and hence the Li-test is pursued.
Li is completely and rapidly burned in a fully convective, low-mass object, once the core temperature reaches about 3 million degrees. The time it takes for a contracting pre main sequence star (or some brown dwarfs) to reach this temperature is mass-dependent. It takes about 20 million years at 0.2 solar masses, about 120 million years at 0.075 solar masses (roughly the boundary between stars, and brown dwarfs that will never become hot enough to fuse hydrogen), and if the object is below 0.06 solar masses, it never gets rid of its Li. The physics of this process is well understood and hence it has become a very useful and trusted tool for mass and age estimation.
At the same time as this happens, the surface temperature and luminosity are also diagnostic of mass, but in an age dependent way. An M7 spectral type is on the brown dwarf limit at about 120 Myr, but a similar spectral type in an older and more luminous object would mean it was more massive.
Historically then, the Li test was applied to low-mass objects in the Pleiades cluster (see Rebolo et al. 1996), which has an age of about 120 Myr. Objects with spectral types cooler than M7 at that age should be brown dwarfs. This was confirmed by finding Li in their atmospheres.
The Li test is not a completely unambiguous indicator of whether an object of unknown age is a star or a brown dwarf. Both stars and brown dwarfs can have spectral types M6.5-L2, but a star would be older at the same spectral type. If you have measured the luminosity (or spectral type) of an object and know whether it has lithium or not, then this provides a constraint on both mass and age. The diagram below, from Basri et al. (1998), is I think as clear as it can be made.
The more-or-less diagonal curves are the tracks along which low-mass stars and brown dwarfs of the labelled mass evolve (getting cooler with age). The diagonal dashed region shows the area of the temperature versus age plot where Li is expected to be depleted. I have labelled the equivalent spectral types. The stippled (dotted) area marks where the presence of Li guarantees that you have a brown dwarf with mass $<$ 0.075 solar masses.
Thus an M6.5 object with Li could either be a brown dwarf or a star younger than about 140 Myr, but an M6.5 object without Li is a star. An M8 dwarf with Li is a brown dwarf with an age less than 200 Myr, but an M8 dwarf without Li could either be a more massive brown dwarf with age just over 200 Myr or an older low-mass star. At spectral type L0-L2 then an object with Li will be a brown dwarf if its age is less than 500-1000 Myr, but could still be a brown dwarf with mass above 0.06 solar masses with an age just a little older than this, or an even older low-mass star. Objects cooler than L2 are always brown dwarfs and always have Li - but this test may fail at much cooler temperatures (cooler than abut L8) because the Li atoms form LiCl and LiOH molecules, the atomic Li I line at 6708A disappears from the spectrum, and the molecular features are too weak/confused to be identified.
A similar diagram can be made with luminosity on the y-axis and actually, that is more secure, since Li depletion as a function of luminosity is much easier to
predict than the behaviour of effective temperature, which depends a lot on the details of the atmosphere. Of course to measure a luminosity you need the distance to the object though.
Nowadays, the presence of Li in a low mass objects atmosphere is more likely to be used as an age, rather than mass indicator. For example, if one sees Li in an M7 object, then it is likely to be younger than 120 Myr (and vice versa). The prediction of Li depletion, particularly as a function of luminosity or spectral type is thought to be more secure than predicting the position of such an object in the HR diagram. This forms the basis of the
lithium depletion boundary age determination method for clusters.
To sum up.
An object with a spectral type of M8 or cooler with Li is a substellar brown dwarf.
An object with Li but a warmer spectral type may be a brown dwarf if it is old enough.
An object younger than $\sim 150$ Myr without Li is a star.
An object cooler than about spectral type L2 is a brown dwarf.