The higher spectral resolution grating would reduce the spectral range. Besides that, would a higher resolution grating reduce the signal per pixel? I thought I heard someone mentioned this to me a while back and I cannot remember the details. I think it would make sense because a higher resolution grating would spread the light out more, but I am not convinced either way. Can anyone provide any insight?
You are correct that using a more dispersive grating will spread your signal out more on the detector. Thus if you have a source with a set flux per unit wavelength interval, then a more dispersive grating decreases the wavelength interval per pixel on the detector and thus decreases the flux per pixel on the detector. This in turn can reduce the signal to noise per wavelength interval because there is a fixed "readout noise" associated with each pixel. Additionally, if your detector is of a set physical size then you can get less of the spectrum on the detector if you use a higher dispersion.
In general terms you would aim to accumulate your spectrum so that it is not readout noise limited. That is, the noise associated with your signal strength in each pixel (basically the square root of the number of detected photons) should significantly exceed the readout noise. To achieve this, you would need to observe longer with a higher dispersion grating, and longer than just the factor by which the dispersion had increased. This may not be practical (e.g. maybe you need to observe something for longer than it is above the horizon!) or it may be that exposures longer than say an hour (in my experience) are so badly affected by cosmic ray hits that they are no longer useful.
Thus you are trading off the additional information you get from the higher resolution spectrum with a smaller spectral range (which could actually be finessed using a cross-dispersing echelle spectrograph for a single object) and a lower efficiency of observation in terms of time taken to get to a given signal to noise per wavelength interval.
tl;dr: 1) High resolving power can come from high order, and order overlap can be a problem. 2) If your application involves measurement and analysis of a continuum spectrum then the last thing you want to have is a deep null or strong slope in grating efficiency in your spectroscopic wavelength range of interest, because it will forever call your calibration into question.
Gratings by themselves don't have "resolution" proper, that's a property of a complete spectroscopy system (slit, optics, detector, etc.).
However gratings can have something called resolving power which depends on how many grooves are illuminated by the instrument:
Resolving Power: The resolving power of a grating is a measure of its ability to spatially separate two wavelengths. It is determined by applying the Rayleigh criteria to the diffraction maxima; two wavelengths are resolvable when the maxima of one wavelength coincides with the minima of the second wavelength. The chromatic resolving power (R) is defined by R = λ/∆λ = nN, where ∆λ is the resolvable wavelength difference, n is the diffraction order, and N is the number of grooves illuminated.
which is just a measure of how widely they spread stuff out without consideration to how useful it is given a complete system.
An analogy might be a low quality f = 1000 mm telescope with an f = 4 mm eyepiece and a 3x Barlow used at sea level soon after sunset. Your magnification will definitely be 750x but your useful magnification might be a heck of a lot less.
The equation for resolving power in that tutorial is
$$R = \lambda / \Delta \lambda = n N$$
where $n$ is the grating order used and $N$ is the number of grooves of the grating illuminated.
Why is $nN$ important? We can think of a grating as a method to interfere beams of different path lengths, a bit like dishes in a large array are used to do the same thing.
We know that the resolution of an array depends on how large the array is divided by the radio wavelength.
We also know that the diffraction resolution of a telescope depends on the number of wavelengths that fits into one diameter, but a telescope has a curved imaging system and a flat array like a radio telescope array or a grating rely only on integer differences in path length and let Huygens allow them to interfere.
Why would someone choose a lower
resolution resolving power grating over a higher one when performing spectroscopy?
Besides the spectral range argument in the question, there's the problem of order overlap.
One way to boost the resolving power of a grating is to operate it in a higher order $n$. The highest resolution spectrometers often use Echelle gratings which are operated at perhaps $n$ = 10 to 50.
The price you pay for that can be gleaned from the grating equation
$$\sin \theta_i - \sin \theta_o = n \lambda / d$$
with focus on the $n \lambda$ part. For example the $n$ = 9 diffracted order of 500 nm will fall right on top of the $n$ = 10 diffracted order of 500 nm, and your sensor will not be able to distinguish them.
note: @ProfRob points out that one can use an order-sorting filter and that's common in spectroscopy. If your visible light can let UV through and your sensor can respond to it, that's a pretty common instance.
Blaze angle and Wood's anomalies
Without going into too much detail, while a given diffraction grating's dispersive behavior follows strict and easy to use mathematical equations, its efficiency behavior as a function of wavelength can be a mess.
You may have a high dispersion (and perhaps resolving power) grating that has some pathological behavior at a wavelength of interest and a low dispersion grating who's "sweet spot" happens to be just where you want to look.
If your application involves measurement and analysis of a continuum spectrum then the last thing you want to have is a deep null or strong slope in grating efficiency in your spectroscopic wavelength range of interest, because it will forever call your calibration into question.
Why? Because calibration is always challenging and these anomalies (especially when sharp) will have angular dependences, meaning how well you fill the entrance angle of the spectrometer might affect efficiency.
You don't want to open that can of worms if you can help it.
Just a quick schematic example from https://www.horiba.com/en_en/diffraction-gratings-ruled-holographic/
This is not a great example, but you can see that even at the same dispersion the Wood's anomaly (that dip in the cartoon efficiency spectrum) is in different places for different gratings, and the overall efficiency vs wavelength is different as well.