I have heard that James Webb will see the first stars that our universe produced. Can I assume that we may see galaxies that are so young that all of the stars in them are population III?
According to Rydberg et al. (2013) "Detection of isolated Population III stars with the James Webb Space Telescope", the answer for non-lensed population III stars appears to be no. For lensed stars, the detection would be difficult but the expected number of detections is very small:
To detect even one 60 M⊙ Population III star when pointing JWST through the galaxy cluster MACS J0717.5+3745, the lensing cluster with the largest Einstein radius detected so far, the cosmic star formation rate of Population III stars would need to be approximately an order of magnitude higher than predicted by the most optimistic current models.
Prospects may be somewhat better for detecting population III galaxies. These would likely be small (dwarf) galaxies in low-density regions, as larger galaxies likely became rapidly polluted by supernova ejecta. From Zackrisson et al. (2012) "Detecting gravitationally lensed Population III galaxies with the Hubble Space Telescope and the James Webb Space Telescope":
We find that the ongoing HST survey Cluster Lensing And Supernova survey with Hubble (CLASH), targeting a total of 25 galaxy clusters including J0717, potentially could detect a small number of pop III galaxies if ∼1 per cent of the baryons in these systems have been converted into pop III stars. Using JWST exposures of J0717, this limit can be pushed to ∼0.1 per cent of the baryons. Ultradeep JWST observations of unlensed fields are predicted to do somewhat worse, but will be able to probe pop III galaxies with luminosities intermediate between those detectable in HST/CLASH and in JWST observations of J0717.
(The designation J0717 is an abbreviated form of MACS J0717.5+3745 referred to in the Rydberg et al. paper.)