The English word "summer" means the season of the year that is associated with higher temperatures and shorter nights. There is no official "first day of summer" and different groups of people take different conventions.
One possible convention is to take "June, July and August" as summer, so the first day of summer is June 1st. This is the convention taken by the Met Office in the UK, and roughly corresponds to the warmest temperatures in the UK.
(The reason that the warmest temperatures are not around the solstice is nothing to do with astronomy, it is because the surface takes some time to warm up, so there is a lag between the longest day and the highest temperature)
Another possible convention is to take June 21st to Sept 20th as "summer". This fits the solstice and equinox and still roughly corresponds to the warmer days of the year in the Northern Hemisphere. This is the convention in many modern calendars.
Another possibility is to take the "cross-quarter days" (named in Gaelic Samhain, Imbolc, Beltane, and Lughnasadh) So summer would be from Beltane/May day to Lughnasadh/Lammas day: May 1st - August 1st. This matches the shortest nights, but generally, May is cooler than August in the UK, so is not consistent with summer meaning "warmest season of the year".
The big point here is that, there is no official definition of summer.