On the one hand an object spans a smaller angle the farther away it is, as expected. On the other hand, due to the expansion of the Universe and the finite speed of light, very distant objects were closer to us when they emitted the light we see today. At that time they spanned a larger angle.
Any point in a galaxy emits light in all directions; the light emitted from a galaxy's edges in the particular direction where your eyes happens to be, defines how large you perceive it, i.e. the angle you measure the galaxy to span. If the Universe did not expand, you would measure the angular size of a galaxy as the angle between the two green lines here:
However, the Universe does expand, so when the "green" rays arrive at the point we were located when they were emitted, we are no longer there. Luckily, the galaxy also emitted rays in the direction where we are now, so that's what we see, illustrated here by the blue lines:
The galaxy is now far away. In a "normal" world, we would see it as small, illustrated by the red lines below, but since we perceive its size by looking along the blue lines, we see it as larger:
Here is an animated version:
The above explanation is a little simplified because expansion not only occurs along the line of sight, but also perpendicular to it. Hence the rays will not take straight lines toward us, but somewhat curved. Moreover, if the Universe is not geometrically "flat", but has a negative or positive curvature, this will also affect the exact path of the rays.
The exact turnover — the threshold between "looking smaller because far away" and "looking larger because closer in the past" — depends on the expansion rate history of the Universe, as well as on the way that light propagates in the Universe, which in turn depends on the densities of the various constituents of the Universe. The interrelationship of these quantities is given by the Friedmann equation.
However, for all viable cosmologies, you will find that the turnover point around a redshift of $z \sim 1.5$, i.e. roughly $9.5\pm0.25$ billion years ago.