I am sorry this question is probably silly for professional astronomers, of which I am not one.
This question is by no means silly. Your question is a common one about cosmology (the study of where the universe came from, how it is evolving and what its fate will be). The media often butchers these concepts horribly, resulting in a lot of confusion (out of all scientific information, they seem to have the hardest time reporting cosmology accurately). Your inquisition is definitely a good thing.
When astronomers say early universe was small, do they simply mean "the part of the universe which corresponds to our observable universe, was small"?
Well, they're usually referring to the entire Universe. In my final paragraph, I explain what this implies for the observable universe.
If it is infinite, then it would seem that it had to be infinite in infancy as well, just very dense. In fact, it seems "in the limit", it had to be infinite even at Big Bang.
You're closer to the truth. When we talk about the expansion of the Universe, we're really saying that space is being created between all matter.
As you mentioned, the Universe may be infinite. It is not like a ball, but rather like a flat grid, and its "expansion" just means that the distances between objects on the grid are getting larger. In essence, more space is being created between the objects. That's what we mean by expansion — that objects are moving away from each other, since more space is being created between them. Below is a gif I've made to demonstrate this:
A more useful way to describe this is to say the grid is expanding — that space itself, as a coordinate system, is growing. As an analogy, imagine are walking your dog. Suddenly, the ground begins expanding between you. You and your dog will separated and continue receding away from each other.
So the same thing is happening with our universe. The grid is in fact growing, and objects are being swept away with it.
OK, now that we've gotten the core concepts down, I'll introduce one more bit of terminology. The "scale factor of the Universe" refers to how much the Universe has expanded, compared to now. For example, if in a billion years the scale factor is 3, that means that every object in the Universe is 3 times farther from each other compared to now. If the scale factor 700 million years ago was 0.8, then everything was closer by a factor of 0.8 at that time. By definition, the scale factor is 1 right now.
So, if the Universe is expanding now, we'd expect it to be smaller as we look further back in time — i.e. the scale factor would be less. General relativity predicts the scale factor to be zero at 13.8 billion years ago. This would mean that every object would be zero times its current distance from us — in other words, there would be no space.
If you think a Universe without space is impossible, you're correct. We apparently have a contradiction. In GR, you can't have a spacetime with zero space.
Our modern physical theories work fine up a few fractions of a second after the moment of contradiction, and our observations do agree with the idea of an extremely dense early universe. However, our theories break down as we try to model the Universe at earlier and earlier times, until they no longer prove accurate, preventing us from explaining the most interesting moment.
This is why the moment of the Big Bang is one of the biggest mysteries in cosmology. Theories like quantum gravity have arisen to try to explain the conditions near the Big Bang, but none are sufficient as of now.
I often hear at lectures that immediately after the Big Bang, the universe was small, say, the size of grapefruit or something like that.
Indeed, the problem stems from the ambiguity when one says "universe". In this case, they're referring to the observable universe, which is actually spherical. The observable universe was indeed much smaller near the time of the Big Bang, compared to its radius now.
This is because its radius actually depends on our Universe's scale factor*, which means that at the moment GR predicts the scale factor to be zero, it also predicts the size of the observable universe to be zero.
Obviously that can't be the case, since as we've explained above, it shouldn't be possible for the scale factor to be zero. However, we can say with reasonable confidence that the observable universe was likely the size of a grapefruit at one point, if not smaller (although "grapefruit" seems an arbitrary choice for comparison. I can't actually find the paper that first uses this analogy, so what they originally meant is a bit unclear).
*Measuring distances is actually a bit tricky in cosmology; in some cases, we want to talk about distances or motion of objects while neglecting the Universe's expansion. To save you the need to learn a lot of terminology, I'm right now taking into account the expansion of the Universe when talking about the observable universe's size. The observable universe also grows due to factors besides the Universe's expansion, i.e. light from further and further galaxies reaching us.
