Your hypothesis is correct: it is the tidal interaction between neighbours which generates spin.
Think of your proto halo (galaxy, stellar disc etc) as an ellipsoid (set by it's inertial tensor). Should the matter around it not be spherically distributed, it will apply a torque onto that ellipsoid and make it spin. The assumption in cosmology is that even though the primordial angular momentum you mention may be null, tidal interaction between neighbouring proto structures is sufficient to allow then to exchange (hence acquire individually) angular momentum, while preserving a zero sum.
A Reference on this subject is given by this paper
This diagram may enlighten the issue: it represents the tidal torquing of the blue ellipsoid by the tides represented by the pink ellipsoid.
PS: the expression spin up is unfortunate it seems because it is in fact spin conservation at the later stage of collapse of the proto halo which makes it rotate faster while preserving it spin (= total internal angular momentum).