The brightness of light received from a light source (or an object that reflects light) is inversely proportional to the square of the distance.
So if an astronomical object A which reflects light from the Sun back to Earth orbits at a distance of two AU from the Sun and an idental astronomic object B orbits at twice that distance, or at 4 AU, how much light will Earth get from the two objects at their oppositions?
Since object B is twice as far away from the Sun as object A, it's surface receives one quarter as much light from the Sun as object A. Since object B will be three tiems as far from Earth at opposition as object A will be at opposition, its reflected light will be one ninth as bright. So together, those two factors will make the light that Earth gets from object at its opposition one thirty sixth, or 0.02777, as bright as the light that Earth gets from object A at its opposition.
So the farther away from the Sun an astronomical object is, the less bright its reflected light will be as seen from Earth.
And if an object C has a higher albedo, or is more reflective than object D of the same size, it will seem brighter at the same distance.
The brighter a solar system object is as seen from Earth, the sooner it will be noticed and discovered. The dimmer a solar system object is as seen from Earth, the longer it can remain undiscovered.
Thus the undiscovered objects in the solar system should be the ones which appear fainter as seen from Earth.
They may be fainter and dimmer as seen from Earth because their albedo is lower than objects at the same distance. Or maybe because they are much smaller than other objects with the same albedo and distance. Or maybe they are very large objects with high albedos but are very faint because they are very far away.
Over time, astronomers discover dimmer and dimmer objects in the solar system, objects which have darker surfaces, or are smaller, or are much farther from the Sun.
So the largest solar system objects likely to be discovered in the future are likely to be very far fromt he Sun and the Earth and thus appear many times dimmer than objects of equal size in the inner solar system.
You may have heard of the hypothetical Planet Vulcan, inside the orbit of Mercury, once used to explain certain problems with the orbit of Mercury. It is no known that Vulcan could not possibly exist. It would have been discovered long ago if it did exist.
None of these claims has ever been substantiated after more than forty years of observation. It has been surmised that some of these objects—and other alleged intra-Mercurial objects—may exist, being nothing more than previously unknown comets or small asteroids. No vulcanoid asteroids have been found, and searches have ruled out any such asteroids larger than about 6 km (3.7 mi). Neither SOHO nor STEREO has detected a planet inside the orbit of Mercury.
The largest solar system objects discovered in recent times have all been beyond the orbit of Neptune, and have been rather small. Only a few of the largest ones have been large enough to classify as dwarf planets, and they are all smaller than Eerth's moon.
It is possible that there may be one or more as yet undiscovered planets in the outer solar system. But since there are limits on how bright those planets could be and remain undiscovered, there are limits on how large and/or close they could be.
As of 2016 the following observations severely constrain the mass and distance of any possible additional Solar System planet:
An analysis of mid-infrared observations with the WISE telescope have ruled out the possibility of a Saturn-sized object (95 Earth masses) out to 10,000 AU, and a Jupiter-sized or larger object out to 26,000 AU. WISE has continued to take more data since then, and NASA has invited the public to help search this data for evidence of planets beyond these limits, via the Backyard Worlds: Planet 9 citizen science project.
Using modern data on the anomalous precession of the perihelia of Saturn, Earth, and Mars, Lorenzo Iorio concluded that any unknown planet with a mass of 0.7 times that of Earth must be farther than 350–400 AU; one with a mass of 2 times that of Earth, farther than 496–570 AU; and finally one with a mass of 15 times that of Earth, farther than 970–1,111 AU. Moreover, Iorio stated that the modern ephemerides of the Solar System outer planets has provided even tighter constraints: no celestial body with a mass of 15 times that of Earth can exist closer than 1,100–1,300 AU. However, work by another group of astronomers using a more comprehensive model of the Solar System found that Iorio's conclusion was only partially correct. Their analysis of Cassini data on Saturn's orbital residuals found that observations were inconsistent with a planetary body with the orbit and mass similar to those of Batygin and Brown's Planet Nine having a true anomaly of −130° to −110° or −65° to 85°. Furthermore, the analysis found that Saturn's orbit is slightly better explained if such a body is located at a true anomaly of 117.8°+11°
−10°. At this location, it would be approximately 630 AU from the Sun.
New moons of the four giant planets are still being discovered, but the recent discoveries are all of tiny objects, less than 10 kilometers in diameter.