Update for clarity:
For the visible part of a standalone galaxy, the stars can all be measured to orbit that galaxy's core. So if you wanted to measure the furthest extent at this simplified level, it would be very easy. The problem is that there is a lot of mass which is not stars, and which is dark matter. Some of it is so far out, it may be impossible to calculate what it is orbiting.
If you read the paper, you will see that they chose a limit beyond which the effect of the negligible mass further out was not relevant to this calculation.
A galaxy can be thought of as a flattened sphere, as regards mass. Stars, at least for spiral galaxies, tend to lie in a plane, but there is mass orbiting the common centre in all planes. So there is no start point, there is just a need to decide how far out we want to say the galaxy extends.
For the Milky Way and Andromeda, a decision has been made that is the same for both (the exact decision doesn't appear to have been published for this paper, but as long as it is consistent, then the relative masses will be correct)
For galaxies that are colliding or near to each other it is much more difficult - do you calculate which galaxy a star belongs to by the direction of its motion? Stars may swap from one to the other. From Wikipedia's Milky Way page:
Surrounding the Galactic disk is a spherical Galactic Halo of stars and globular clusters that extends further outward, but is limited in size by the orbits of two Milky Way satellites, the Large and the Small Magellanic Clouds, whose closest approach to the Galactic center is about 180,000 ly (55 kpc). At this distance or beyond, the orbits of most halo objects would be disrupted by the Magellanic Clouds. Hence, such objects would probably be ejected from the vicinity of the Milky Way.
tl;dr - it's a pretty arbitrary decision :-)