It would be rather difficult to make negative mass consistent with the rest of physics and not have a bunch of weird stuff happen. According to general relativity, mass causes a curvature in spacetime, and objects follow that curvature. So whatever effect negative mass has on spacetime, and thus on positive mass objects, would also be the effect it has on the movement of negative mass objects. If negative masses repel positive masses, then they also repel negative masses.
Now, you may think that in the gravitational force formula, the force is the product of the two masses, so two negative masses would give a positive force (where "positive" means "directed towards the other mass"). However, there's also the fact that acceleration is force divided by mass, so a positive force divided by a negative mass would give a negative acceleration. As the answer given in James K's comment notes, this means that given a negative and positive mass, both of them would experience a negative force, which for the positive mass would mean repulsion, but for the negative mass would mean attraction.
If we try to make an atom out of particles that are identical to protons and electrons, except that they have negative mass, then the electrical force would still be a positive force as it is with normal atoms, but this would result in the particles being repelled. The strong force, which normally keeps nucleuses together, would instead push them apart. So coming up with a way of making matter, rather than just isolated particles, would be rather nontrivial. You could have a negative-proton and a negative-position (that is, a particle that's like an electron except its charge is positive and its mass is negative) or a negative-anti-neutron and a negative electron making an analog of a neutron atom, but a helium atom has four particles in its nucleus, so this tactic wouldn't work for them. Even single baryons would be problematic, since the force that holds quarks together would be repulsive. Moreover, since like charges would be attracted and opposite would be repelled, particles would segregate into clumps of all-positive charges and other clumps of all-negative.