The mass of Vesta was determined by observing its effect on 197 Arete. The model used is that Vesta will cause the eccentricity and semi-major axis to change. By observing the asteroid before and after their encounter, and determining the asteroid's orbital elements before and after, one can find the angle of deflection. Model the motion as a hyperbolic encounter and one can find the combined mass. Since 197 Arete is much smaller than Vesta, this gives a good estimate of the mass of Vesta.
The mass of asteroids can also be found by their perturbations of Mars, providing there are sufficiently accurate observations. Viking radar data (with an accuracy of 7m) allowed for good estimates of the mass of the major asteroids. The Dawn mission to Vesta and Ceres made very high accuracy measurements of the gravity field around them and so their mass.
The math involved is essentially just Newtonian mechanics. You know the positions and velocities before the encounter. There is an energy conserving encounter with a force of $$F = \frac{m_1 m_2 G}{r^2}$$ Finding the angle of perturbation allows you to find the reduced mass $$\frac{m_1 m_2}{m_1+m_2}$$