I certainly don't know the details of these kinds of calculations, but as my thought is a bit too long for a comment I'll write it up as an answer.
If you measure the flattening of a planet due to rotation (e.g. by measuring its rotation period) and the gravity (measured while keeping the orbiter at constant height above the planet), then you have everything you need to calculate the moment of inertia I/(MR2) using Radau's approximation (not the most informative link, but I couldn't find a better one). After that, it's a matter of taking a model for a planet, e.g. a two-layer model with a core and a mantle, which has free parameters x, the fractional core radius, and f the difference in density between both layers of the planet. Compare your measured moment of inertia with the model and you've got x and f for that planet, which allows you to calculate the mass of the core (if you also have the total mass of the planet from gravity measurements). The figure below shows the two-layer model and the measurements for some solar system bodies. The value 0.4 on the y-axis is the value for a homogeneous planet. From that you can deduce that Ganymede is a very differentiated planet. Hope that helps a little bit.

A note on how this graph was created: first of all, this isn't my graph, I took it from my lecture notes. The moment of inertia factor for a homogeneous body is calculated as
$$I = \frac{8\pi}{3}\int_0^R \rho r_0^4 dr_0 = \frac{8\pi}{15}R^5 $$
$r_0$ is the mean radius of a shell within the planet (because planets generally aren't round).
Then you take a simple two-layer model for a planet where $\rho(r) = f\rho_0$ for $0\leq r \leq xR$ and $\rho(r) = \rho_0$ for $xR < r \leq R$, where $R$ is the radius of the planet.
The moment of inertia factor is then
$$I/MR^2 = 2[1 + (f - 1)x^5]/5[1 + (f - 1)x^3] $$
which is the curve that you see in the figure.