The latter.
To astronomers, a metal is any element that is not hydrogen or helium, because these elements together constitute most of the elements in the Universe, by far.
This means that, in many circumstances all other elements can be neglected, at least to first order.
By mass, H and He account for some 74% and 24% in the present-day Universe, respectively, while the next-most abundant elements are on the <1% scale. And because these elements are also heavier, if you consider the abundance by number, H and He account for roughly 99.9% of all atoms.
Lithium, on the other hand, only account for a few billionths of all atoms. Here's an overview of the abundances in the Solar System, in terms of number densities, i.e. not mass densities:
Credit: Wikimedia Commons user 28bytes, under C.C.-by-S.A.-3.0.
The reason for the zig-zag pattern is that, in general, elements with an even number of protons are more stable than those with an odd number, as discussed in this post.
Note that the $y$ axis is logarithmic, i.e. each tick mark denotes a ten-fold increase in abundance. The axis is arbitrarily scaled to $10^6$ silicon (Si) atoms, so the Si dot is at $\log A \equiv 6$. Oxygen, for instance, is at $y \simeq 7.15$, so it is $10^{7.15-6}\simeq14$ times more abundant than Si.
Are we wrong?
The terminology used by astronomers may be considered wrong by other physicists and chemists. But why do we use the term at all? To distinguish some of the elements from other elements, based on their properties, I'd say. In astronomy, it very often makes sense to consider the elements of the Universe as divided in three:
- hydrogen, which is most of the mass and most of the atoms, and which provides most of the energy in stars,
- helium, which adds significantly to the mass, lowering the mass fraction of hydrogen, and
- all other elements, which together help cooling the gas (so it can condense), form dust grains (altering observed spectra), have a very roughly equal amount of protons and neutron (so $A\simeq2Z$), are formed in other processes than the Big Bang (except for a small part of the lithium), etc.
In physics, you may be more concerned with the material's ability to conduct electricity and hence consider sodium a metal until the pressure becomes too high, after which you call it a non-metal.
In chemistry, you may be more interested in the materials ability to form bonds, and hence do not consider arsenic and antimony metals.
So, I think you can say that there isn't a formal definition of the term "metal" — it depends on the context. From Wikipedia:
Around 95 of the 118 elements in the periodic table are metals (or are
likely to be such). The number is inexact as the boundaries between
metals, nonmetals, and metalloids fluctuate slightly due to a lack of
universally accepted definitions of the categories involved
Apart from these practical reasons, there are also historical reasons, as discussed in this answer by Rob Jeffries.