James K's answer is great, I just want to offer a few definitions:
Any mass $M$ — whether a be point mass like a planet or an extended mass like a galaxy — has an associated gravitational potential $\Phi(\mathbf{x})$. This is defined as the energy needed to bring a unit mass from the point $\mathbf{x} = \{x,y,z\}$ to infinitely far away from $M$.
The escape velocity $v_\mathrm{esc}$ is defined at the point $\mathbf{x}$ to be the velocity an object needs to achieve just enough kinetic energy to overcome the depth of the potential "well" and get infinitely far away from $M$, without having to spend more energy propelling itself.
You ask "What is the escape velocity calculated relative to?" You can thus say that it's calculated from any point you wish, relative to infinitely far away (in practice, just far enough that gravity is no longer dominated by the galaxy, planet, or whatever, but is dominated by other objects).
The potential can be a hard concept to visualize, especially in 3D, but you often see it depicted in a 2D analogy as a depression in an otherwise flat surface. You can then think of $v_\mathrm{esc}$ as the kick you need to give a ball to make it roll up the well, without rolling back. Here's an illustration of the combined potential of Earth and Moon (from Wikipedia):
Mathematically, you calculate the escape velocity as
$$
v_\mathrm{esc}^2(\mathbf{x}) = 2|\Phi(\mathbf{x})|.
$$
Outside of a spherically symmetric object (e.g. Earth), this evaluates to $v_\mathrm{esc} = \sqrt{2GM/r}$, where $r$ is the distance from the center of the mass. For an extended mass (e.g. the Milky Way), the expression becomes more complicated and depends on the density profile $\rho(\mathbf{x})$ of stars, gas, and, in particular, dark matter. That is, the exact distribution of its component matters.
By observing the velocities of various objects (stars or luminous gas clouds) in a galaxy, we get the rotation curve and can then map the density profile. Given $\rho(\mathbf{x})$, we can then calculate the potential by solving Poisson's equation:
$$
\nabla^2\Phi = 4\pi G \rho,
$$
where $\nabla^2$ is a mathematical description$^\dagger$ of how the steepness of the potential changes from place to place.
For galaxies (or more specifically the gravity-dominating dark matter halo in which the galaxy resides), it often it turns out that, to a good approximation, $\rho(\mathbf{x})$ is given by a so-called NFW profile, but many other profiles are seen as well. The exact density profile can of course only be known if the exact mass and position of every single star, planet, or even gas particle, is known, but on large scales, the average profile is an excellent approximation.
And once you have the potential, you have the escape velocity.
$^\dagger$$\nabla^2$ (also written $\Delta$) is called the Laplace operator, and is defined as the divergence ($\nabla\cdot$) of the gradient ($\nabla$), where the gradient itself is $\equiv \{\partial/\partial x, \partial/\partial y, \partial/\partial z\}$.