Most, and perhaps all, stars form in stellar clusters. Most likely, the sun formed in a open cluster. In clusters the density of stars is quite high, therefore close encounters are common. So, I think it would be quite likely for such close encounters to disturb Oort clouds enough to send comets out of their host systems. Some small fraction of these comets would be recaptured by other stars within the open cluster.
In addition, it is thought that in the early period of planetary disk formation, there were many more planets than today, but they often collided with each other, often sending pieces out of the stellar system. Moreove, many small planets and asteroids were thrown out by gravitational interactions with the large planets (see Tremaine). These 'debris' would be roaming interstellar space. Occasionally, one could fall into any random stellar system.
The odds that we have any asteroids or comets from any particular nearby star is low, unless it happened to have been from the Sun's birthplace. We do not have the ability to determine which of the nearby stars are from the Sun's open cluster, but the Gaia Satellite may make that possible soon. And, if we explore the solar system long enough, we may find exo-solar asteroids among the hordes of asteroids.