Encounters between small planetesimals and the larger core of planets (especially Jupiter) transfer momentum between the two. But momentum is conserved. Momentum is:
p = m v$p = m v$, where p$p$ is momentum
So, smaller planetesimals gain more velocity than a larger planet core, for the same momentum exchange. Thus, smaller planetesimals are preferentially boosted to escape the planet, either to be scattered outward to the outer system, or inward to the sun. Some will have the right velocity and aggregate with to core.
Since smaller planetesimals get a larger boost than larger ones, say 1000x rather than 20x (just an example, not real numbers), then they are much more sensitive to initial velocities. Thus a much smaller range of initial velocities will have that exactly right velocity to be captured by the planet, as either a satellite or aggregating with the planet. In the case I just made up, 1000/20 i.e. 50 times narrower. Thus, for an equivalent distribution of velocities, the smaller planetesimals in this example will be 50 times less likely to be captured.
As for dust, beyond it will eventually clump together into larger rocks due to electrostatic attraction, a lot will be sucked into the growing core, or scattered outwards. But it will exert a net drag on the core (see the Grand Tack hypothesis) and gain momentum, and through velocity, energy (a.k.a. heat). And this energy will be averaged out through elastic collisions. Eventually, the dust will either aggregate into larger bodies, sucked into the planet, or scattered into the sun or outwards from the solar system.