Disclaimer: I have a Machine Learning/Computer Vision background, not Astrophysics, so please be a bit patient with me. If you feel like this question is too far out there it would be great if you could direct me to meta or some bulletin board/mailing list that you think is a better fit. :)

Important: I am talking about "datasets" in a machine learning / computer vision manner. I do not mean "the entirety of SDSS data" but rather a common set of inputs and (for supervised learning) associated outputs for a specific problem.

I have been looking for a while now to tinker with some problems in Astronomy/Astrophysics and I keep finding the same kind of advice when it comes to datasets:

Go to SDSS/GalaxyZoo, they provide downloads for their data.

Now, while that is true, none of these datasets are anywhere near as accessible as datasets in Computer Vision. "Over here" a dataset is mostly a bunch of (huge) tar files with jpeg images and some .mat or .csv for annotations. The process of getting started is pretty straigth-forward: unzip, load annotations into memory, start sciencing (or usually, applying wild heuristics). Obviously I simplify slightly, but you get the idea.

Now, to stay with the example of GalaxyZoo, the object coordinates, annotations and computed features are conveniently presented as a csv, fits or voTable file. But to get the actual visual data (if I understand this correctly) I have to, for each annotated galaxy...

  • determine a bunch of .fits urls from the coordinates in the annotations (one per band and galaxy) and download them (~12MB/file)
  • download the containing "tiles" from the SDSS servers (in each of the bands I'd like to work with)
  • extract the patch containing the galaxies (with the aptly named sExtractor)
  • possibly apply some non-trivial corrections/transforms to the image patch

I tried following along this procedure with a semi-randomly chosen paper from arXiv for which there is code available here and without wanting to make any statement about said paper/code, this process is not comparable to the simple workflow I described above, and I would probably download some TB of data, were I to use the entire dataset, of which only a tiny fraction would actually be useful.

The resulting dataset only occupies a few GB of space, and has a very simple nature: ca 50000 images, with associated labels -1 or 1, for stars or galaxies. The data preparation pipeline, while one might also want to optimize it for this task, has next to nothing to do with the "real" work being done in the paper, and so having the reader go through those steps makes little sense to me.

So my questions are:

  1. Did I understand the workflow correctly? Maybe I'm just following the wrong kind of paper, or I really just overlooked a big red "Download Here!" button? Otherwise:

  2. Did anyone go through these steps and make the resulting dataset available? Or are there good reasons not to do so? From the size of the sExtractor manual I gather that the choice of parameters in this processing step might significantly influence the outcome of any downstream method. So...

  3. Are there any sane/agreed-upon default settings for the preprocessing pipeline? While for any given method, applying extra "magic" in the preprocessing might make sense, having a default set of parameters would make dataset creation and a bunch of other things much easier, no?

I am sorry if this is a naive newcomer question. And for the long post. :P As a meta, I would add tags "dataset" and "programming" or some such, but I lack the reputation.

  • $\begingroup$ Looks like TRQ (the real question) is "how do I properly process the datasets I found?' Do you want to edit the title to reflect this? $\endgroup$ – Carl Witthoft May 11 '17 at 12:38
  • $\begingroup$ Assuming that the answer to 1 and 2 are "yes" and "no", respectively, that would be true and question 3 is going in that direction. But given that, I would probably rephrase it more like "How come the dataset distribution in astronomy is so complicated on the receiving side?" It puts an awful lot of work on the person trying to replicate an experiment, and on the data hosting servers, no? $\endgroup$ – black_puppydog May 11 '17 at 13:56
  • $\begingroup$ By the way I do realize that for an insider, this question may sound like a lazy newcomer whining, and in that case I'd really like to hear why this is not happening. I can imagine that people have good reasons to not simply use a dataset from somewhere else, instead making sure the preprocessing steps best match their method. But for replication specifically, a big download would likely be the best, especially for datasets of the "we use a random sample of size N from catalog C" kind. $\endgroup$ – black_puppydog May 11 '17 at 13:59
  • $\begingroup$ Could you clarify what kind of data you are looking for? I suspect the problem stems from the fact that you are looking for a dataset that astronomers don't usually use (i.e. selection by "unusual" criteria), so there's no need, from an astronomer's perspective, to have the access mechanism you seek. I tried to replicate your problem and went to SDSS website, but found it very straightforward to find FITS images for the first object that came to mind, the Coma cluster. How to proceed from there depends strongly on the kind of analysis one wants to do, so there's no universal answer for that. $\endgroup$ – Alex May 13 '17 at 14:44
  • 1
    $\begingroup$ @black_puppydog: Please explain what you mean by "entire datasets". I'm sure you have something in mind, and I'm sure it's very different from what I am thinking of. If you want to work on every image that SDSS has ever observed, of course you'll have to download every image they have ever observed. If you want, for example, all galaxies that are members of a galaxy cluster beyond redshift of 0.1, that requires detailed analysis that noone has done yet, and you'll have some objects missing as well as false positives. So, please clarify what you mean by "entire datasets". $\endgroup$ – Alex May 13 '17 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.