While the eye is terrible at determining apparent magnitude or brightness of stars due to our adaptive iris, it's perfectly capable of discerning relative brightness between two objects in the same visual field.
Also, the ancient Greeks could discern six magnitudes of star brightness by realizing that brighter stars look bigger in the night sky. The word 'magnitude' means 'bigness' or 'size'.
Magnitude - Wikipedia
This wasn't the most accurate system, but it was good enough to be able to discern about six different magnitudes. The system dates back about 2,000 years, to either Hipparchus or Ptolemy. So there's no question that ancient greeks had at least a rough categorization of stars by brightness.
Now, could they tell when a planet was closest? Easily. The difference in magnitude between closest and farthest distance from the Earth for most of the planets Ptolemy could see was very large. For example, Mercury at its brightest as seen from Earth has an average apparent magnitude of -1.89, and at its faintest, +5.93. A change in magnitude of that amount would be easily discernible by the human eye. Venus differs by about two magnitudes. Mars almost five magnitudes. Jupiter about 1.3. Saturn is the only one with a difference of less than one magnitude, making it uncertain if the ancient greeks could see that difference.
Calculating Apparent Magnitudes of Planets
If the ancients couldn't determine the absolute magnitudes of the planets, they could certainly compare them to the magnitudes of known stars and determine if they were brighter or dimmer. Given the huge magnitude swings of the inner planets, determining when they were closest would have been relatively easy, within some margin of error.