Or is it possible to spot the changing brightness (magnitude) of a planet with a naked eye
The context of the question is this: If I'm not mistaken Ptolemy was able to say (using observation alone) that the retrograde motion of a planet happen to be when it was closest to earth (that behavior was consistent with his epicycles model).
So I have my doubt with respect to the accuracy that this kind of observation could be done at all.
While the eye is terrible at determining apparent magnitude or brightness of stars due to our adaptive iris, it's perfectly capable of discerning relative brightness between two objects in the same visual field.
Also, the ancient Greeks could discern six magnitudes of star brightness by realizing that brighter stars look bigger in the night sky. The word 'magnitude' means 'bigness' or 'size'.
This wasn't the most accurate system, but it was good enough to be able to discern about six different magnitudes. The system dates back about 2,000 years, to either Hipparchus or Ptolemy. So there's no question that ancient greeks had at least a rough categorization of stars by brightness.
Now, could they tell when a planet was closest? Easily. The difference in magnitude between closest and farthest distance from the Earth for most of the planets Ptolemy could see was very large. For example, Mercury at its brightest as seen from Earth has an average apparent magnitude of -1.89, and at its faintest, +5.93. A change in magnitude of that amount would be easily discernible by the human eye. Venus differs by about two magnitudes. Mars almost five magnitudes. Jupiter about 1.3. Saturn is the only one with a difference of less than one magnitude, making it uncertain if the ancient greeks could see that difference.
Calculating Apparent Magnitudes of Planets
If the ancients couldn't determine the absolute magnitudes of the planets, they could certainly compare them to the magnitudes of known stars and determine if they were brighter or dimmer. Given the huge magnitude swings of the inner planets, determining when they were closest would have been relatively easy, within some margin of error.
Variable star observers can obtain 0.1 magnitude accuracy by comparing the brightness of a star with 2 nearby stars of known magnitude; one a bit brighter and one a bit fainter. Most estimates are done telescopically with all 3 stars in view at the same time. The accuracy is much worse when doing this by naked eye and red stars are more difficult. So the ancient Greek astronomers or anyone else could tell when the planets were brightest/faintest.
Modern amateur astronomers do regularly track the brightness of naked eye variable stars. The American Association of Variable Star Observers (AASVO) has a light curve generator. Try it out for Algol and Betelgeuse. Use select magnitudes choose just Vis (top left). https://www.aavso.org/LCGv2/
