2
$\begingroup$

I am using this index page to download GDR2 data files.

http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/

But it keeps giving a different number of links. Sometimes it shows 16,000 files, sometimes 31,000 and the max I have seen is 61,237. What is the correct number I should download? GAIA GDR2 documentation does not list the total number of files available.

Interestingly, if the index lists less than 60k files, it still has the closing /html tag so it would appear I have a complete index file, but the contents are not complete.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Using curl, I got 61240 lines total, but that includes some headers, so I think 61237 is correct and your browser may just be timing out. In particular, the last file on my list is "GaiaSource_999922404314639104_1000172126596665472.csv.gz", which seems correct, since the first number increases with each file and seems to finally "wrap around" with the last file. $\endgroup$
    – user21
    Commented Apr 29, 2018 at 3:54
  • $\begingroup$ Thanks, I did more tests and 61237 seems to be the correct number. Add this as an answer and i will accept it. $\endgroup$
    – Ags1
    Commented Apr 30, 2018 at 10:04
  • $\begingroup$ Thanks. I'm not really into the points thing, but feel free to post it as your answer. $\endgroup$
    – user21
    Commented Apr 30, 2018 at 17:04

2 Answers 2

Reset to default
4
$\begingroup$

There are 61,234 data files in the directory. There are three additional (non-data) files:

MD5SUM.txt     
_citation.txt  
_disclaimer.txt

If you include those, the total number is 61,237.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Proofs: curl -s 'http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/MD5SUM.txt' | grep 'GaiaSource_' | wc -l for the terminal or [...document.querySelectorAll("a")].filter((el) => /GaiaSource/.test(el.href)).length for the browser console (F12) in a modern browser. $\endgroup$
    – try-catch-finally
    Commented Dec 1, 2019 at 13:37
0
$\begingroup$

barrycarter provided the answer:

Using curl, I got 61240 lines total, but that includes some headers, so I think 61237 is correct and your browser may just be timing out. In particular, the last file on my list is "GaiaSource_999922404314639104_1000172126596665472.csv.gz", which seems correct, since the first number increases with each file and seems to finally "wrap around" with the last file. – barrycarter Apr 29 at 3:54

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .