The R in that equation is the distance from the star to observer, not the star radius. The light emitted from the star is distributed uniformly on a sphere of radius R, and when the light arrives to the Earth, that sphere will have a radius equal to the distance Earth-star.
Therefore, the second relation for the two fluxes is about the apparent magnitudes (which describe the brightness of an astronomical object observed from Earth),
$$m-m_\odot = -2.5 \log F/F_\odot$$
The first relation is alright. The absolute magnitudes are related with the luminosity of the star (the overall energy flux emitted by the star) and they are not dependent on the distance to the observer.