This is something that crops up a lot in Astrophysics and Astronomy, so here is the general approach.
Get your equation into SI units. Thus
$$M = \frac{5v^2R}{G} = 7.5\times10^{10} v^2 R,$$
with $M$ in kg, $v$ in m/s and $R$ in m.
Now if you wish to replace $v^2$ by $(v/1000)^2$ (which is velocity expressed in km/s), you would simultaneously multiply the numerical constant at the front by $1000^2$. Similarly, to replace $R$ by $R/3.09\times 10^{22}$ (which is $R$ expressed in Mpc), then you have to multiply the numerical constant by $3.09\times10^{22}$ to leave everything unchanged. This then leaves your result still expressed in kg. To express it in solar masses, simply divide by a solar mass of $2\times 10^{30}$ kg.
Thus
$$\frac{M}{M_{\odot}} = \frac{7.5\times 10^{10}\times 10^6 \times 3.09\times10^{22}}{2.0\times 10^{30}} \left(\frac{v}{1 {\rm km/s}}\right)^2 \left(\frac{R}{1 {\rm Mpc}}\right)$$
$$\frac{M}{M_{\odot}} = 1.16\times10^{9} \left(\frac{v}{1 {\rm km/s}}\right)^2 \left(\frac{R}{1 {\rm Mpc}}\right)$$
Finally, check your result makes sense. Galaxy clusters are of order a Mpc in size, have velocity dispersions of hundreds of km/s and consists of hundreds of galaxies with total mass of $\sim 10^{14} M_{\odot}$ or so. This seems to check out - usually if a mistake has been made you will be many, many orders of magnitude out.
