A book suggestion: The Big Bang by Joseph Silk
The main text is descriptive only, yet deep enough into the topic to be recommended for my third year cosmology course as an undergraduate. The math that supports the concepts is covered in the Appendices and may be the best indication on how much you need to learn and in which areas to support your interests.
Math is just the language that you use to do the science. The more math you know, the more types of analysis you can do. Some simple math has far reaching implications, as in Hubble's Law where you can get an estimate on the age of the universe by taking the inverse of Hubble's parameter. Likewise, Special Relativity only requires high school algebra, but the implications are mind-blowing.