You can do a rough 0th order estimate:
The Sun has about $1360 W/m^2$ on the Earth's surface and a brightness of $\approx -26$ magnitudes. Every 5 magnitudes corresponds to a factor 100 in brightness or every magnitude corresponds to a factor of $^5\sqrt{100}$. The moon has $\approx -13$ magnitudes. Thus it is about 13 magnitudes less bright than the Sun:
$$P_{moon} = \frac{1}{^5\sqrt{100}^{13}}P_{Sun} = P_{Sun}\cdot 6.3\cdot10^{-6} = 8.5mW/m^2$$
If you do that for a planet with -2 magnitudes (so about Venus or Jupiter give or take a magnitude), you are left with $340nW/m^2$ and for the most distant planet, Neptune with $\approx 7$mag only $8.5\cdot10^{-11} W/m^2$.
All values above are the power per square meter. For a telescope with 5 inch of unobstructed aperture you have about 1% of these values ($\pi\cdot (0.0254\cdot5)^2/4$), neglecting losses in the optics and reduced aperture for the sake of secondary mirror mounts etc.