You are referring to what is known as proper motion, which is the apparent motion on the sky due to the relative motion of the Solar System and the stars.
To measure an accurate parallax you have to observe for long enough that you can disentangle the parallax motion, which is of course modulated on a 1 year cycle, from the proper motion of a star, which produces an approximately linear, monotonic drift of the star's position with time.
This is perfectly illustrated in this graphic produced by HST/NASA, which shows the track on the sky taken by Proxima Centauri (a nearby star with a large parallax and a large proper motion). The small, periodic wiggles are caused by the parallax shift, whilst the linear trend in the path is due to proper motion. The two are easily separated if you observe for at least 18 months with several precise position measurements during that time.