I make telescopes and telescope mirrors. You're asking a hard question, since it depends on so many things. But it's an interesting project, so I'll try to help as much as I can.
The hard limit for performance is aperture, or diameter of the primary optics. Bigger aperture = higher hard limit for performance. On a nanosat you only have room for so much aperture, so performance will necessarily be limited.
Secondly, you'd need some kind of compact design. A reflector for sure, but not newtonian. Some kind of Cassegrain variant, you can make those pretty short and stubby. Unless you make it so that it "unfolds" in space, but that's going to be tricky.
Also, the whole optical stack has to be very rigid and must hold its shape through launch and while in space. The optical system must remain collimated at all times.
Finally, you probably have some weight limits. That will make it harder to create a rigid system.
One measure of performance is the resolving power, which indicates the angular size of the smallest details resolved by the telescope. The relationship is linear. An aperture of 100 mm provides a resolving power of 1 arcsec. An aperture of 200 mm provides a resolving power of 0.5 arcsec. And so on.
Telescopes operating at ground level are limited by atmosphere in several ways.
One is the so-called "seeing", or air turbulence. This blurs the image and reduces the effective resolving power. Since small telescopes have less resolving power to begin with, they are less affected. A 100 mm aperture is unaffected most of the time. A 200 mm aperture is affected most of the time.
Your telescope will be above the atmosphere, so the effects of seeing will be zero. It will operate at full resolving power all the time.
Another way the air influences telescope is via light pollution and other forms of luminescence and reflected light from the air. This reduces contrast and "erases" the faintest objects from the image; bright objects such as planets are not affected at all.
Your telescope will not be affected by light pollution, so its performance on faint objects (such as galaxies or nebulae) will always be 100% of the maximum for that aperture.
Bottom line: you are going to have a pretty small telescope, but operating completely free of the effects of the atmosphere.
Since it's operating in microgravity, the optics will not deform under their own weight, so you can make them quite thin and lightweight.
The instrument will operate under extremes of temperature. You will need some kind of dynamic focusing to compensate for expansion / contraction. Be careful, with some catoptric designs such as Cassegrain you also need to pay attention to the distance from primary mirror to secondary elements, etc. Using carbon fiber for the skeleton would reduce expansion / contraction a great deal, and would also provide excellent stiffness per unit of weight.
Quartz mirrors could deal very well with huge temperature variations, but quartz is expensive to process to optical precision. Borosilicate glass (the material used for most amateur telescopes) should also work well; Supremax 33 for example is cheap and fairly good. It depends on the optician who will make the mirrors. If you find someone who could give a discount on quartz, go for it; otherwise use borosilicate and don't worry about it.
You will also need some kind of inertial elements to change the orientation of the telescope in space. That's not a trivial task.
Bottom line: you begin with a performance budget limited by aperture. That's the ideal performance. Then real world issues will push real performance below that level: collimation errors, temperature shifts, optics quality, focusing errors, etc.
Performance of a real instrument is affected by everything.