Is it possible for there to be a dense enough nebula to form stars outside of any galaxy? Does a galaxy have a minimum size to produce stars? Or could you have a few dozen stars clustered together by themselves?
Yes, stars can form outside galaxies if the conditions are right. An impressive example is D100, a galaxy that is moving through a cluster so fast that the ram pressure from the ambient gas forces galactic gas out of it leaving a long tail. That tail is dense and cold enough to allow star formation, and there are newly formed clusters in it.
In principle star formation can happen anywhere where there is dense, cold gas that can collapse gravitationally. Stars can sometimes help by forcing gas together by radiation pressure. There are intergalactic gas clouds, but they are usually too hot. Dark matter halos are not great at capturing hot gas, but intra-cluster gas is losing energy from Bremsstrahlung so over time some of it would accumulate there.
Space is large, so no doubt there is some star formation happening in corners where gas clouds have cooled off remotely from galaxies. After all, that was how stars and galaxies once started.
It's quite possible for stars to form outside of galaxies, typically in environments where large amounts of gas have been stripped from a galaxy. This usually involves either a tidal interaction with another galaxy or the intracluster medium (ICM). In the latter group are a set of peculiar galaxies sometimes dubbed "jellyfish galaxies". Gas, dust and stars are being stripped away by ram pressure from the high-speed collision with the ICM, and some of this gas can then collapse to form star clusters in the gaseous tail behind the galaxy.
For some candidate intergalactic star formation regions, it's unclear whether the young, blue stars found in blue knots were ejected with the gas or formed from it, but it's still reasonable to assume that star formation will eventually take place in the cold gas clumps.
Notable examples of stars forming in gas stripped by ram pressure include:
- D100, as mentioned by Anders Sandberg (Cramer et al. 2019)
- The Comet Galaxy (Cortese et al. 2011)
- ESO 137-001 (Sun et al. 2007)
In the other group, gaseous structures ejected during galaxy interactions can also form stars, under the right condition. The one notable example I'm aware of is the Leo Ring (Michel-Dansac et al. 2010), but I assume that other cases have been proposed.