This can be done in a number of ways. There is a "theoretical" approach, where the transmission and reflection characteristics of all the components are measured in the lab and put together to get an overall, wavelength-dependent efficiency. This isn't used much apart from in system design.
An on-site method is to measure the spectra of spectrophotometric standard stars, using a very wide slit, so that all the light is captured. These standard stars have measured fluxes as a function of wavelength that can then be used to calibrate the response of the spectrometer. A correction must also be made for atmospheric extinction, that might involve observing standards at the same elevation as the target, or at a range of elevations to actually assess the extinction. The history of how spectrophotometric standards were established is long and complicated.
Standard software packages take the spectrum of the standard star to obtain a response function that can be applied to other spectra. Algorithmically, the observed spectrum is rebinned into intervals that match a table of calibrated standard star fluxes in the same intervals. The standard star calibration data is divided by the rebinned spectrum, taking account of the ratio of exposure times and any estimate of relative extinction, to get the conversion in flux per binned count. Any observed spectrum can be multiplied by (an interpolated and possibly smoothed version) of this to put a flux scale on an observed spectrum. A good description is included in the help file for the old IRAF package sensfunc.
It is rarely the case that spectra have an accurate absolute flux calibration. Usually, stars are observed using slits or fibers to improve spectral resolution or improve wavelength calibration. In such circumstances, a variable amount of light actually enters the spectrograph, associated with turbulence in the atmosphere (not a problem for HST/JWST) and prevents accurate absolute flux calibration.
Sometimes all you want to do is remove the shape of the wavelength response. This can also be done by observing the spectrophotometric standard stars, even if the absolute flux level is uncertain. The absolute flux level could then be established if you have broad-band photometry of the source at a similar wavelength range (providing the source isn't variable).