One way to figure out if one (or both) of the objects is a black hole, neutron star, white dwarf, or other compact object would be to try to measure its mass. For example, a neutron star and a white dwarf are both compact stellar remnants. However, there is one decisive factor that determines which type of stellar remnant a progenitor star will become: the remnant's mass.
The Chandrasekhar limit ($\sim1.39M_{\odot}$) is the maximum mass of a white dwarf; the Tolman-Oppenheimer-Volkoff limit ($\sim1.5-3.0M_{\odot}$) is the maximum mass of a neutron star. The large uncertainty in the latter means that it's not particularly useful for identifying compact objects that could be neutron stars or low-mass black holes.
So if you know the mass of one of the objects - and let's assume that it's well within the mass range of one of the classes of stellar remnants - you might be able to figure out which type it is. How would you measure the mass? Well, you could study the orbit of its companion star to try and determine the stellar remnant's effect on its orbit.
Another way would be to study gravitational waves emitted by the system. These waves can only be emitted under certain circumstances - for example, in a system of binary neutron stars - see, for example, the Hulse-Taylor binary system, also known as PSR B1913+16. The power radiated as these waves, as well as the orbital decay, depend on the masses of the objects. While detecting gravitational waves is an incredibly difficult task, there are several detectors planned or already in operation, such as eLISA and LIGO.
You could also attempt to look for other effects that characterize certain stellar remnants. For instance, a certain pattern of radio waves would characterize a pulsar (white dwarfs and black holes are not normally strong radio sources). However, if the objects are not luminous, this may not be effective.