The Oort cloud is disturbed. This is thought to be one of the ways we can get long period comets. It is also depleted of objects over time and it is likely that only a fraction of the original objects in the cloud remaining.
I guess what you really mean is why isn't the Oort cloud completely removed on a short timescale? The reason is that not only does an interloper have to get close enough, it has to be there long enough to strip satellites from the Sun.
The basic physics here is that when a star passes close to the Sun it attracts both the Sun and it's satellites towards it. What matters for whether it will strip a satellite from the Sun is the difference in attraction the Sun and the satellite feel, known as a tidal force. The tidal force depends on the mass of the interloper and the inverse cube of its proximity. A back of the envelope equation might be a condition for tidal stripping would be
$$G\frac{M m}{r^3}a > G\frac{M_\odot m}{a^2}\ , $$
where $M$ is the mass of the interloper, $a$ is the semi-major axis of the satellite and $r$ is how close the interloper gets to the Sun.
This rough equation tells us that stripping may occur if
$$ r < \left(\frac{M}{M_\odot}\right)^{1/3}a\ .$$
This would suggest that all that is required is for another star to enter the Oort cloud for this to happen.
But the treatment above assumes that $r$ and $a$ don't change. Whilst this might be a good approximation for $a$, it isn't for $r$. Interlopers will have a distribution of relative velocities with respect to the Sun, but a typical value might be about 20 km/s. That means the interloper spends a relatively short time in the regime where $r \sim a$ and is certainly moving too fast to capture a satellite from the Sun. Again, a rough calculation - the escape speed from the Sun at 50,000 au is about 200 m/s. The acceleration that another 1 solar mass star passing 50,000 au away provides is $2\times 10^{-12}$ m/s$^2$. For this acceleration to add 200 m/s to a satellites velocity takes of order $10^{14}$ s (or 3 million years). But moving at 20 km/s an interloper will travel 50,000 au in just 10,000 years. Thus the interloper may not be in the vicinity of the Solar System for long enough to accelerate the satellite significantly.