The Vogt-Russell "theorem" says that the structure of a star is uniquely determined by its mass and the distribution of chemical elements within its interior.
To answer your question, you need to decide what you are holding fixed. A star of a given mass and composition has a fixed radius. If you increase the mass you increase the radius. If you fix the mass and make to star more metal-poor, then you decrease the radius (but not by very much for M-dwarfs).
The surface temperature is determined by the luminosity and radius. The luminosity is governed by the central temperature. An increase in mass, increases the central temperature and greatly increases the luminosity. The radius also increases, but not by enough to compensate, so the surface temperature also rises. At a fixed mass, a decrease in metallicity does not greatly affect the central temperature or luminosity, so a decreasing radius means that the surface temperature increases.
The basic answer is that the equations of stellar structure determine the structure of the star and there is a limit to how much handwaving you can do.
In recent years it has been realised that the Vogt-Russell theorem is not the end of the story, especially for M-dwarfs. Their radii and surface temperatures also appear to depend on how fast-rotating and magnetically active they are. This is due either to suppression of convective heat flux by interior magnetic fields or the blocking of flux by starspots at the surface (e.g. Jackson et al. 2018 - work I am involved in). Both of these effects make the stars bigger and cooler at a fixed mass.