The most likely scenario is that an asymmetry in the supernova explosion imparts momentum to the proto neutron star at its core. The issue is not settled.
A recent study by Verbunt et al. (2017) models the
speed distribution of young pulsars as the sum of two Maxwellian distributions, one with an average speed of $\sim 130$ km/s and the other with an average speed $\sim 520$ km/s. About 30% of the population are in the low speed population and only a few percent have speeds below 60 km/s.. A plot of the (model) distribution is shown in Fig.9 of that paper.
Why do some pulsars move so fast? Conservation of momentum. Pulsars are neutron stars with mass about $1.5 M_{\odot}$, born at the heart of core-collapse supernovae. A 500 km/s speed corresponds to a kinetic energy of about $4\times 10^{41}$ J and a linear momentum of $1.5\times 10^{36}$ kg m/s, but during the supernova, approximately $10^{46}$ J is made available when the core collapses from about the size of the Earth to a ball of neutrons 20 km across. Thus only a tiny fraction of this energy needs to be given to the neutron star to accelerate it to high speeds.
Most of the energy in a supernova is released as almost massless neutrinos from the core. In order of magnitude terms, the neutrinos carry away a linear momentum of $\sim E/c = 3.3\times 10^{37}$ kg m/s. If there were a small asymmetry in how these neutrinos escaped - i.e. more in one direction than another, then conservation of momentum would demand that the core moves in the opposite direction. To match the observed linear momentum of fast moving neutron stars would demand
a directional asymmetry of neutrino emission of $\sim 5$%, (which seems unlikely and may require magnetic fields of $10^{12}$ T - Nardi & Zuluaga 2001 ). A better idea might be that in the initial stages of the collapse, the core material is opaque to neutrinos and it could be that there is an asymmetry in the way the neutrinos break out of the absorbing material (e.g. Nordhaus et al. 2012).
Alternatives are that there are instabilities in the final phases of nuclear burning, just prior to the core collapse. These instabilities cause the core to wobble back and forth and when the supernova happens, the core just happens to be travelling one way or another. There have also been suggestions that asymmetric radiation from the pulsar itself could gradually accelerate it to the speeds observed.