Does coal burn at room temperature? Most people would say "no". For an oxygen molecule to react with an atom of carbon in the coal, a certain amount of energy is required to break the bonds joining the atoms together. Once these bonds have been broken, then new bonds can form releasing energy (and the energy released is more than the energy required to break the bonds - it is an exothermic reaction)
But at room temperature most of the oxygen molecules are moving too slowly; they don't have enough energy to react, so no reaction occurs. Of course not every molecule of oxygen has the same speed, some have more energy. A very very small proportion might have enough energy to react. Yet even so, you would not say that coal burns at room temperature.
Well much the same is true in a Brown dwarf. The protons need a lot of energy to overcome the electrostatic repulsion of two positively charged protons. There is a distribution of energies among the protons and there must be a very very small proportion of protons that do have sufficient energy but just as you could leave a piece of coal from millions of years, Brown dwarfs will never convert more than an utterly neglible amount of their mass their mass into helium
Image from Review of the Universe
The reaction rate drops preciptously below about 3 million K. A key reason is that just as with coal, as the temperature rises, and the random reaction rate increases, at a certain point the energy released, which gets transferred to nearby particles, causes the reaction rate to increase further. A positive feedback loop. For coal, or other flammable this when we say it has caught fire. For dwarf stars, this is when we say that the core is supported by fusion and gravitational collapse has been halted: the mark of a true star.
So while there will, inevitably, be very rare fusion events, below about 3million K no self sustaining p-p chain reactions can occur, and the core of the brown dwarf will continue to collapse under gravity.