or is it that the definition of infinity used here excludes such events like there are an infinite number of odd numbers but 2 isn't in that set?
The universe has a finite age, about 13.7 billion years. So there has not been time enough for infinite events in our past light-cone.
There are rather few events in mainstream physics that are universe ending. Closed universes eventually implode, and if certain forms of dark energy exist they can cause a Big Rip where the universe expands to infinity in a finite time. But obviously observers cannot look back on them in their past. This is also true for other local disasters like false vacuum decay.