So, imagine, we are getting really good at off-earth mining at a larger scale and in a larger timespan, we continue to mine thousands and thousands of asteroids.
First off, that is a problem for our children's children's children to solve. It is not our problem. The quantity of materials mined from space so far is a bit over 380 kilograms, the 382 kilograms returned from the Moon by the Apollo missions, the 300 grams of materials returned from the Moon by automated Soviet spacecraft, 7 particles of interstellar dust returned from comet Wild 2 by the Stardust spacecraft, and 1500 particles of dust returned from asteroid Itokawa by the Japanese Hayabusa spacecraft.
For the foreseeable future, there is very little stuff in asteroids that is worth anything on Earth. There is no point in mining commodity metals in space and bringing that material down to Earth. Doing so would be a money-losing proposition. Mining precious metals in vast quantities would turn those expensive precious metals into worthless commodity metals.
One thing that might be of value on Earth is helium 3, but that is so very, very rare that bringing it down to Earth will have zero effect on the Earth's mass. This of course depends on making fusion viable; that's been cited as being twenty years in the future since the 1960s. Current estimates are well beyond 20 years.
The foreseeable future (at most 50 years from now; anything beyond that is science fiction) of asteroid mining is using those materials mined in space in space. That opens the door to extracting substances that are extremely common on Earth. The same economics that dictates that mining precious metals in space doesn't make sense economically means that sending consumable up into space doesn't make much sense economically. For now, the most valuable substances to be mined in space are volatiles such as water and methane. This assumes a sizable space infrastructure that needs those substances.
Wouldn't that affect the gravitational balance of our star system at some point, or is the gravitational influence of an asteroid belt negligible?
The asteroid belt accounts for about a billionth (one part in 109) of the mass of the solar system. The gravitational influence of the asteroid belt is negligible.