tl;dr Conservatively 10%, realistically 25%, optimistically 60%.
I assume that by the "observable-by-us Universe", you mean not the theoretically observable Universe, which is given by the distance light has had the time to travel since the Big Bang, but the part of the Universe where we may practically see galaxies and other objects, not including the cosmic microwave background.
Current redshift record
The currently most distant, spectroscopically confirmed object is the galaxy GN-z11 (Oesch et al. 2016), which has a redshift of $z=11.1$ and hence a distance of just over $d=32$ billion lightyears (Glyr). The volume inside the sphere with us in the center and GN-z11 on the surface is $V = 4\pi d^3/3 \simeq 140\,000\,\mathrm{Glyr}^3$.
How far will James Webb see?
Predicting the redshift record for James Webb is not easy, since it depends on the unknown physical conditions of even younger galaxies. The important quantity is their (surface) brightness which depends not only on intrinsic properties such as their star formation rate, dustiness, and compactness, but also on the properties of the intergalactic medium, in particular to which degree it is (re-)ionized and hence able to transmit the light from the galaxies. Moreover, we also don't know how many there, so estimates rely on extrapolating their distributions of luminosities (luminosity functions) from lower redshifts.
Rather conservative predictions for Webb's redshift record are around $z\simeq13\text{–}16$
(Mashian et al. 2016;
Williams et al. 2018;
Mahler et al. 2019;
Behroozi et al. 2020), corresponding to an age of the Universe of 250–300 million years after the Big Bang (whereas GN-z11 is seen ~400 Myr after the BB), and a distance of 33–34.5 Glyr.
These estimates hence correspond to volumes of $\simeq 150\,000\text{–}170\,000\,\mathrm{Glyr}^3$, i.e. an increase of up to roughly 10–25%.
However, sometimes we're lucky and line up with a massive cluster of galaxies acting as a gravitational lens which magnifies the light from distant background sources by factors of tens, hundreds, or even thousands. This may potentially allow James Webb to see some of the very first galaxies, predicted to have formed at $z\sim20\text{–}30$, 100–200 Myr after BB. I don't know of any peer-reviewed papers doing any serious estimates, but NASA routinely cites Webb's predicted record as "200 (or 250) Myr, possibly even 100 Myr" af the BB (e.g. here). If we really discover a galaxy 100 Myr after the BB, that would correspond to $z=30$, and a distance of 38 Glyr, i.e. a volume of $225\,000\,\mathrm{Glyr}^3$ which is 60% larger than our current "observable-by-us Universe".
The extension of our probed volume may not seem like a lot. A better way to understand the significance of this is to think about the age of the Universe at that time: It is likely that we will make the unknown epoch of the Universe half, or maybe even a quarter, of today's value.
For comparison, the volume inside the region from which the cosmic microwave background was emitted at $z=1100$ is $390\,000\,\mathrm{Glyr}^3$, while the volume of the (theoretically) observable Universe is $413\,000\,\mathrm{Glyr}^3$.
The figure below is pretty much to scale:
