How far outside the visible spectrum could an optical telescope feasibly operate?
I guess it depends what you mean by an optical telescope.
However, if you just mean any telescope that can record images at visible light wavelengths, then it is possible to use these (with the appropriately cooled instruments) to make observation at so-called mid-infrared wavelengths of $\sim 20$ $\mu$m.
See for example on the Gemini telescopes https://www.gemini.edu/sciops/instruments/midir-resources
The background noise at such wavelengths is extremely high - the telescope and sky look hot at these wavelengths - so these instruments have largely been superseded by space observatories like Spitzer and WISE.
Almost all large "optical" telescopes are routinely used to make observations out to $2.4\ \mu$m (the K band).
It depends on the telescope.
If the telescope is fully reflective (no transparent glass elements), it should be able to operate well-beyond visual wavelengths, with its transmission limited primarily on your mirror costing material.
Any telescope with refractive (transparent glass) elements however, will quickly lose performance outside of the visual range due to transmission, but also from how the glass index changes with wavelength, which causes non-visual light to be defocused (an effect which increases the further you get from visual wavelengths).
Generally speaking, the defocused issue of refractive telescopes can be compensated if you have enough focus travel. You will see "chromatic aberration" becoming much more significant very quickly, as telescopes are only designed to correct this over visual wavelengths.
Ultimately, the limiting factor of refractive telescopes is the transmission of the glass in the lenses. Many glasses will transmit reasonably well from 400-2000nm, though some have worse performance and some have better.