The mass of total matter in a galaxy can be measured using the rotation curve of the galaxy. However, how do people determine the mass of luminous matter in the galaxy, as a function of the distance from the center of the galaxy?
The luminosity function would be determined using photometry. Then do people simply assume a certain mass-to-light ratio for the stars in the galaxy and then convert the luminosity function to a mass function?
I assume that by "luminous matter" you mean stars, although quite as often astronomers are concerned with the mass of gas — which could then be the total gas mass, the mass of some particular element or molecule, or, more often, the mass of a particular elements in a particular state (e.g. "doubly ionized") — or it could be the mass of the dust which, although it does absorb light, also emits light.
Various methods exist for each of the cases, and indeed one way to estimate the stellar mass is by assuming a mass-to-light ratio. However, this relation must be established in the first place by measuring the mass and the light of some galaxies and/or stellar regions.
The most common way of deducing the stellar mass is by modeling the "spectral energy distribution", or SED, and fitting the model to your observation. This approach yields, among other things which I'll describe below, the stellar mass based on the observed light.
SED fitting is typically done for the "whole" galaxy, i.e. averaging over the whole luminous region and hence not telling you anything about how the mass decreases with distance from the center. However, it is in fact also possible to carry out this analysis on a region-by-region basis (in principle pixel-by-pixel, although pixels are typically gathered to reduce the noise), and thus estimate how the stellar mass (and other parameters) vary across the galactic disk (see Östlin et al. 2014).
An SED tells you how much flux a galaxy emits in various wavelength bands, and can be thought of as an extremely low-resolution spectrum (the width of such a band could be e.g. 1000 Å, whereas the width of a spectrum wavelength bin could be 0.1 Å).
A galaxy's SED is determined by all its luminous matter. In the UV and visible light, the light is dominated by stars, but the exact distribution will depend on the properties of the stellar population. That is, how many small stars are there relative to how many large ones, and what is their metallicity (distribution). The relative masses, in turn, depend on the initial mass function, which tells you the distribution when the stars are born, as well as whether the stars were born at the same time (a "starburst") or more continuously as time has passed.
The observed light differs from the emitted light if there is dust in the galaxy, which scatters and absorbs light. The exact way that dust alters the SED depends on its composition and distribution. The dust itself will glow in the infrared, often dominating the SED there. At even longer wavelengths you have e.g. synchrotron radiation.
Assuming some values for all the parameters discussed above, a synthetic galaxy spectrum can be calculated, and hence a model SED. Varying all the parameters then gives you different model SEDs, and the one that fits your galaxy's SED the best may then be taken to be representative of the galaxy, yielding not only stellar mass but also dust mass, metallicity, and more.
The figure below (from Bothwell et al. 2013) shows an example: The gray line is the synthetic model spectrum that best fits all of the colored dots, i.e. fluxes in various filters. The parameters used to produce that gray line is then reported as the physical properties of the observed galaxy.
And yes, the are degenerate parameters, and various models yield somewhat different answers. Nevertheless a quite good estimate for the galaxy's stellar mass can be obtained. You can also get an idea of the galaxy's redshift, but here the uncertainty is large enough that a redshift obtained in this way is not considered well-established; that's why you need to confirm it spectroscopically.
So, the dust mass can also be estimated using SED fitting. But if you can get a "real" spectrum, then you can investigate its composition and measure the temperature (assuming a Planck spectrum) and then from the total brightness infer its mass to a higher precision.
Spectral lines are also used to obtain gas masses. You know how much light is emitted from a given element in a given state so the integrated luminosity of a spectral line tells you the total mass of the gas emitting.
Since a spectrum is normally obtained for the full galaxy, again this approach doens't tell you how the mass decreases with radius. But with so-called integral field spectroscopy you get a spectrum for each pixel in the image, although the spatial resolution is much lower than for a typical image.
