1
$\begingroup$

Our data set has $10^4$ data points, but has a long baseline and many gaps.

If we bin the data, there would be $10^8$ data points ($[t,\rm {value}]$), but only about 1% are non-zero values.

How to improve the detection efficiency?

Is a multi-threading way possible (especially for Lomb-Scargle)?

For example, if I use LombScargle in astropy.stats,

freq, power = LombScargle.autopower(minimum_frequency=0.5, maximum_frequency=1.5, normalization='standard')

there are two problems:

  1. It is slow and easy to get memoryError.
  2. It can't utilize the computing power as much as possible.

So my question is about the efficiency. For the data set above, what is the best method?

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ Question might be a better fit for Computational Science SE. $\endgroup$
    – StephenG - Help Ukraine
    Apr 16, 2019 at 8:45
  • 1
    $\begingroup$ Understanding the Lomb-Scargle Periodogram explains that technique nicely. Would you simply like to know if it can be (or has already been) written in a function particularly conducive to multi-threading? First glance of my (non-expert eyes) suggests that would be really straightforward, but rather than multithreading, I think the calculation is easy vectorized. I think you could ask a more specific, practical question about the technique here (how to set it up with your data) and a more computational question in CompSci as mentioned above. $\endgroup$
    – uhoh
    Apr 16, 2019 at 11:07
  • 1
    $\begingroup$ @uhoh could you suggest some method which can improve the computing efficiency? $\endgroup$
    – questionhang
    Apr 16, 2019 at 13:02
  • $\begingroup$ okay that would definitely be a good, and separate question, asked separately, in Computational Science SE. However, before you ask it, please find a way to explain clearly how you are doing it now. You can't ask how to improve something if you don't show your current method and state its speed. In the mean time, I will take a quick look at what you've added here :-) $\endgroup$
    – uhoh
    Apr 16, 2019 at 13:08
  • $\begingroup$ @StephenG yep that makes more sense. $\endgroup$
    – Inertial Ignorance
    Sep 28, 2019 at 22:49

1 Answer 1

Reset to default
1
$\begingroup$

AstroPy

Jake Vanderplas (Director of Open Software at the University of Washington eScience Institute) has written a Pythonic Preambulations post Fast Lomb-Scargle Periodograms in Python which by definition appears to be a good place to start to learn how to do Lomb-Scargle Periodograms... Fast... in Python.

I have not read it yet in full, but I've found other posts of his to be extremely useful.

He starts out by explaining the implementation in AstroPy which if you are going to be doing Astronomy, is probably a good thing to become familliar with.

SciPy

Alternatively, there is scipy.signal.lombscargle which you can try assuming you have a current version of SciPy installed but want to try something different than AstroPy or want to compare performance or results.

StackOverflow

I found over 40 posts containing 'Lomb-Scargle' in Stack Overflow which is a very good sign!

I notice that you've been active there and know how to ask well-received questions. Since your asking about speed and errors, StackOverflow is your go-to source! However, don't forget to be as clear as possible and quote all error messages and symptoms, and see if you can somehow add a short Minimum Complete and Verifiable Example or MCVE. That's hard when your problem results from a very large dataset I know.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I upload a test data set. See that data link please. The data is a series of photon arrival times. We need to bin the data and then do lomb-scargle or fft. The problem is our low computing efficiency now. We need a faster method. $\endgroup$
    – questionhang
    Apr 16, 2019 at 13:52
  • $\begingroup$ @questionhang thanks, that's great! I recommend two things, 1) wait for someone with specific experience to add another answer here, and 2) consider asking a separate question in StackOverflow as I discuss above. It's really great that you are so responsive to comments and questions. As you know, in SO they can come very quickly! :-) $\endgroup$
    – uhoh
    Apr 16, 2019 at 13:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .