Every day, babies are born and people grow, which makes their respective masses greater. However, this change in mass (should) come out of the food that they consume - it is used as energy and thus converted into this growth. Likewise, building new structures is just a redistribution of mass already on Earth.
Thus, the only way I can see the actual mass of the Earth changing is by meteors that have landed here (increase in mass) and things that leave Earth such as space shuttles and rockets which would be a decrease in mass.
But have I missed anything out? If no meteors crashed into Earth, and we had not yet figured out how to make machines that could fly, would the mass of the Earth remain constant? Or is it somehow loosing or gaining mass? Am I wrong in assuming that the growth of animals doesn't affect the mass?
EDIT: Mass includes atmosphere :)