Given the phase of a planet or satellite, I can find the area illuminated visible to us, if we consider a 2D surface. But how do I find the percentage of illuminated area visible considering it a 3D surface and thus finding the magnitude of the partially illuminated planet/satellite.
In general, you'd have to project the light from the sun (Assuming we're talking about objects in our solar system) onto the 3D surface and project it onto a 2D surface. This can be accomplished via a ray tracing algorithm.
This procedure will give you the fraction of the surface illuminated.
Now if you are concerned about the fraction of the illuminated surface that is reflected in a specific directly -- so that it is "viewable" from that direction -- you have to do the ray tracing in 3D from the source (Sun) to the object and on to the viewer. There packages you can google that will do this type of ray tracing if you can model the objects in 3D...