The answer is partially historically and partially scientifically. Flux, i.e. energy of radiation integrated across a solid angle (steradian) is basically counting all photons coming from a solid angle, while specific intensity (units of specific intensity are $erg/s/cm^2/Hz/sr$), is the energy per frequency along a line of sight! A surface brightness is flux divided by solid angle, i.e. the mean intensity within this solid angle. This evenly smears out all photons in this solid angle.
As a consequence, flux is the preferred unit if one observes a point source, i.e. a source with a solid angle smaller than the angular resolution of the detector. For point sources, the beam size does not matter. If we observe the same point size with two instruments with different beam sizes, they both detect the same flux because they just count all photons coming from the source. A larger beam does not change this quantity. On the other hand, the brightness (mean intensity) of the object goes down with a larger beam size because we average the flux over a larger area.
If you observe an extended (larger than the beam!) object, e.g. a molecular cloud in radio, then surface brightness is the preferred quantity. Assume it is equally bright across its surface, then the surface brightness becomes independent of the beam size because the flux is proportional to the beam size. A telescope with a larger beam (smaller dish) observes a larger flux (always assuming the object is even larger) compared to a larger telescope with a smaller beam. Comparing the two measurements becomes difficult. The surface brightness of both observations is the same though.
Since optical astronomy was primarily observing stars, i.e. point sources, they used flux. Radio astronomers observed the interstellar medium, i.e. extended clouds of gas and dust, so they used surface brightness.