I'd like to add onto what James K said, but I'll try and simplify things too.
A singularity may sound like everything was at a "single point", but that's not what the word means in physics/math. A singularity, loosely speaking, is just a point where a function reaches infinity. It's not a physical object, but an input to a function. As James mentioned, 1/x has a singularity at x=0. At the moment of the Big Bang, several functions have singularities, as I discuss below.
So with that set aside, let's explain the Big Bang. Our modern theory of gravity, general relativity, indicates that the universe is currently expanding (i.e. space is being created between all matter). By that logic, we expect the matter in the universe to be closer together as we look further back in time. Our physical theories indicate that at earlier periods, temperatures were so high that even atoms couldn't form.
Our theories can handle these conditions pretty well, but eventually they start to fall apart. At a point around 13.8 billion years in the past, our calculations predict that the amount of space in the Universe should be zero. As a result, density, gravity and temperature would skyrocket to infinity — in other words, they have singularities at that moment. This moment is known as the Big Bang.
These infinities are a problem, since our theories break down at such extreme conditions. The Big Bang is an unsolved problem that modern physics has yet to explain. It's not the beginning of the universe per se, but rather we do not know what happened around, or before, that moment.