What barriers - of technology, physics and possibly economy (things that would be possible technologically but are simply too expensive) sets the upper bound on quality of telescopes for observation of the sky in visible spectrum - observing surface of distant objects at maximum resolution/detail?
To answer the question reworded as: "What limits the quality of telescopes currently?" The answer is mostly: money.
It used to be: atmosphere. But with advances in adaptive optics, ground-based telescopes are achieving what used to require a space telescope. Plus, we have the technology for space telescopes if we want, like the pending JWST.
So pretty much, it boils down to funding. Who is going to spend the money of the expensive technology to see better and further, when science is being squeezed for funding from all quarters, and astronomers can't put forward a business case for a return on investment like a biotech company can for developing a grass that will result in cows burping less methane.