The Sun and most of the other stars are in the bulging disk of the Milky Way galaxy, but about 1% of the galaxy's stellar mass is in the galactic halo. The halo also includes 50 globular clusters and about 20 satellite galaxies according to Helmi 2008: The stellar halo of the Galaxy. Here is a nice graphic:

Note that the Sagittarius Stream of stars is extremely close to passing the galactic poles. If you were on a planet next to one of these stars as it passed a pole, you would probably have a glorious view of the Milky Way stellar disk.
Bodies in the galactic halo don't necessarily follow the elliptical paths predicted by Kepler, so their orbits may not be consistently perpendicular to the galactic plane. Some stellar streams have extremely odd orbital paths like the Phlegethon stellar stream, referenced in this answer:

There is some speculation that the Halo stellar streams are the remnants of dwarf galaxies the Milky Way has absorbed. Indeed, Liang et al. say that:
the Galactic halo has complicated assembly history and it not only
interacts but also [is] strongly mixed with other components of the Galaxy
and satellite dwarf galaxies