Is it possible to use Photolithography for telescope image sensor?
Sure, in the sense that CCDs and similar devices are already made using photolithography, and have been for decades.
But since you then go on to ask:
Will it be possible to use a chemical image sensor to achieve a nanometer pixel size?
it seems that your focus is on the idea that increasing resolution comes from pixels being as physically small as possible -- which is not actually true. There wouldn't actually be any benefit for image sensors in astronomy to have nanometer-sized pixels.
The measurement scale for images of the sky is angular (radians, degrees, arc minutes, arc seconds) and so is the scale for determining limits of resolution. Typically, we talk about this in terms of the point-spread function (PSF), which describes the blurring imposed by the atmosphere and the optics of the telescope and is usually characterized by its full-width at half-maximum (FWHM). The fundamental (diffraction) limit on resolution for a telescope is FHWM in radians $\approx 1.2 \lambda/D$, where $\lambda =$ the wavelength of light and $D =$ the diameter of the telescope's aperture (i.e., the diameter of the main mirror). For a wavelength of 500 nm (green) and a telescope mirror of 10 m, this means FWHM $\sim 0.01$ arc seconds.
If your telescope is on the ground, it will be looking through the atmosphere, which imposes extra blurring, giving you a PSF with a larger FWHM. Even at the best sites, you are very unlikely to do better than FWHM $\sim 0.5$ arc seconds. (Adaptive optics can correct this for small patches of the sky, but then you still have the ultimate diffraction limit from the telescope diameter.)
The second thing to understand is that the actual image of the sky that is formed on the detector can be scaled up or down (magnified or reduced) using the optical design of the camera (separate from the telescope itself). This is what determines the angular scale of the final image (e.g., arcsec/mm); combined with the physical size of the pixels, this determines the angular size of the pixels in the image (e.g., arc seconds/pixel). This in turn determines how big your pixels are relative to the size of the FWHM, so we can talk about pixels/FWHM as a potential measure of resolution.
Now, at this point you might be thinking, "But surely more pixels/FWHM will always be better!" This is, alas, only true up to a point, because of the Nyquist Sampling Theorem. This tells you that in order to recover the maximum possible spatial information in an image, you need pixels small enough that two of them cover the FWHM. But -- and this is key -- you can't gain anything extra with pixels smaller than that size. (There are practical considerations having to do with square pixels sampling round PSFs, so that one can argue that it's slightly better to have, say, 2.5 or 3 or pixels per FWHM. But there's no gain to be had by going from 2 or 3 pixels per FWHM to, say, 10 pixels per FWHM.)
One way to think about it is this: going from $\sim 2$ pixels/FWHM to 10 or 100 pixels/FWHM (e.g., by making really small pixels) is like taking a blurry photograph and enlarging it: you just get bigger blurs (more pixels per FWHM), not better resolution.
So once you have pixels small enough that -- in combination with the camera optics -- you get $\sim 2$ or 3 pixels per FWHM, you're done: you cannot do any better than that. And since we can build cameras (like that for the LSST) using existing CCDs which achieve this, there's no point in trying to make physically smaller pixels for CCDs.