Why don't we see a change in temperature when the barycenter changes?
The motion of the Sun relative to the Solar System barycentre (SSB) doesn't have much of an effect on the Earth's orbit. As the Sun spirals around the SSB it (mostly) drags the planets, moons, asteroids, etc, with it. It's probably best to treat the Sun's location as fixed, with the SSB spiraling around it, especially when analysing the orbits of the smaller bodies in the Solar System. The Sun's motion relative to the SSB is only significant for the giant planets, but even with Jupiter the effect is small because the solar mass is ~1000 times Jupiter's mass.
I have some info and diagrams of the motion of the SSB here and
here.
However, the seasonal variation in the Earth's climate is not primarily due to the change in the distance from the Earth to the Sun. It's mainly due to the change in the angle of the sunlight.
Earth's mean orbital radius is ~149.6 million km, but it varies between ~147.1 and ~152.1 million km, due to the eccentricity of the elliptical orbit (see below). The change in distance due to the axial tilt is relatively small, eg, it's under 3300 km at 40° latitude.
However, the change in the angle of the Sun at noon has a significant impact. As the Earth revolves around the Sun, the axial tilt (~23.44°) causes the Sun's declination on the celestial sphere to vary between -23.44° to +23.44°. This changes the angle of the sunlight at noon. It also affects the times of sunrise and sunset.
From Wikipedia Effect of Sun angle on climate
Seasonal change in the angle of sunlight, caused by the tilt of Earth's axis, is the basic mechanism that results in warmer weather in summer than in winter. Change in day length is another factor.

One sunbeam one mile wide shines on the ground at a 90° angle, and another at a 30° angle. The one at a shallower angle covers twice as much area with the same amount of light energy.
So when the horizontal angle is 30° (vertical angle = 60°) each unit of area gets half as much energy as when the horizontal angle is 90° (vertical angle = 0°). In general, the intensity is proportional to the sine of the horizontal angle, which equals the cosine of the vertical angle.
This projection effect is also why the Earth is (generally) hottest at the equator and cooler at higher latitudes.
You can test the projection effect with a flashlight and a white wall or sheet of paper, but it can be hard to see, especially with a strong light source. That's because it's hard-wired into our visual processing system, so our brains automatically compensate for it, and interpret such brightness changes as angle data. That is, when we see two planes of the same material at slightly different angles, our brains see them as "really" having the same brightness.
That Wikipedia quote mentioned that "change in day length is another factor". I have some info and graphs about that here.
The distance from the Earth to the Sun does have an effect on the climate, but it's fairly minor. Currently, the Earth is closest to the Sun (perihelion) in early January, around 10 days after the December solstice, during the northern hemisphere winter and southern hemisphere summer. That makes the southern summer a bit hotter than the northern summer. It also makes the southern summer a bit shorter, because the orbital speed is fastest near the perihelion. However, the climate in the southern hemisphere is strongly affected by the strong circumpolar ocean current around Antarctica, which keeps the southern oceans cold all year round. In the southern hemisphere, not many people live at higher latitudes than 40° because it's just too cold, but that latitude in the northern hemisphere is quite heavily populated.
I have some info about the perihelion here and here.
For what it's worth, here's a graph of the distance from the Earth to Sun and to the SSB, for 2022, with a 7 day timestep between the data points.
Plotting script from https://astronomy.stackexchange.com/a/49823/16685
Here's a plot spanning 1700 to 2200 (the same timespan as my Sun-SSB plot in the answer linked above), also with a 7 day timestep.
