I need to extract information from a few columns in ~20k different fits files with python. Each file is relatively small, ~0.2MB. I have been doing this so far with a loop and astropy like this:

from astropy.io import fits

data = []
for file_name in fits_files_list:
    with fits.open(file_name, memmap=False) as hdulist:
        lam = np.around(10**hdulist[1].data['loglam'], 4)
        flux = np.around(hdulist[1].data['flux'], 4)
        z = np.around(hdulist[2].data['z'], 4)
    data.append([lam, flux, z])

This takes for the 20k fits files ~2.5 hours and from time to time I need to loop through the files for other reasons.

So my question is: Can I minimize the time for looping? Do you know of other packages besides astropy that would help me? Or do you know if I can change my algorithm to make it run faster, e.g. somehow vectorize the loop? Or some software to stack quickly 20k fits files into one fits-file (TOPCAT has no function that does this for more than 2 files)? Thanks for any ideas and comments!

  • 2
    $\begingroup$ Does the situation improve with memmap=True ? Could you use from astropy.table import Table ; table = Table.open(file_name) to read them in as Astropy Tables instead and then join them together into one more efficient FITS table ? (Astropy Table docs) $\endgroup$ Nov 10, 2021 at 2:19
  • 1
    $\begingroup$ @astrosnapper Thanks for answering! unfortunately memmap=True doesn't improve the speed. I think I tried astropy.table and I decided to not use it, because I couldn't figure out how to read with it hdulist[0], hdulist[1], hdulist[2] a.s.o. I think it was reading only one of the tables, but my fits files contain 3 tables. I don't know if it matters, but I am running the code on a google colab notebook without utilizing the provided GPU or TPU (they make the code even slower) $\endgroup$
    – NeStack
    Nov 10, 2021 at 12:39
  • 1
    $\begingroup$ Try using multiprocessing and queues - docs.python.org/3/library/… - put() each item in your file list in a queue (call it "inbox"), have a function which works on a single item which it get() out of the inbox, and have it put() the result in another queue ("outbox"). Use multiple processes (see the multiprocessing.Pool class) and you should be able to spread the load across your CPU cores. $\endgroup$
    – Aaron F
    Nov 10, 2021 at 17:41
  • 1
    $\begingroup$ @AaronF Thanks! I will give it a try. Do you think it will also work on google colab - this is the environment in which I am coding this loop. And the data is stored in my google drive $\endgroup$
    – NeStack
    Nov 10, 2021 at 19:17
  • 1
    $\begingroup$ @NeStack you are dominated by file I/O overhead for this many files which is why more CPU, GPU etc isn't helping. Opening and closing many files isn't quick on regular filesystems and while I have no experience of Google Collab, our experience of AWS EC2 with FITS files on S3 is that it's quite a bit slower than even NFS. So effort should be in reducing the amount of I/O needed, particularly if you are re-running this code many times. Tables with multiple HDUs can be read by adding hdu=2 to the Table.open() call doc link $\endgroup$ Nov 10, 2021 at 19:47


You must log in to answer this question.

Browse other questions tagged .