I have light curves for a particular star. I'm able to construct a periodogram of the stars' light curve, a plot of the power of the light curve vs. the signal's period. For a nonperiodic but irregular signal, I'd still expect to see some peaks and valleys on the periodogram just because of random noise. How could I distinguish the non-periodic stars from periodic signals given their light curves and/or periodograms?
There are some nice and well-tested routines to create periodograms and test the periodicities for significance in the astropy.timeseries package, containing the LombScargle class.
This class includes a method for calculating the "false alarm probability", which I think is almost what you are looking for. This gives the probability that you would see a periodogram peak as high as observed if the data had no periodicity at all, at that particular frequency.
To generalise this to whether you would detect a peak as high as observed at any frequency requires some sort of Monte-Carlo or bootstrapping technique, the latter is also implemented in the LombScargle class and can be conceptually understood as replacing each observation point with a value chosen randomly from your list of observations whilst keeping the time the same; then the periodogram is recomputed. Do this multiple times and you build up the distribution of peak heights in the periodogram for data which are essentially free of any periodic signal.
None of the above quite answers the question - what is the probability that the underlying source is periodic, given these data? Nor can it give you the probability that the source is not periodic. Both of these questions need to be answered by simulating data and folding it through your periodicity detection scheme.
These techniques and issues are discussed in detail by Vanderplas (2018) (particularly section 7), which is highly recommended and should be mandatory reading for anyone doing time-series analysis of light-curves.