2
$\begingroup$

There are several files. I think that they all needs to downloaded to have the complete dataset. Is there a particularly good way to download them all in one shot?

The page shows over 100 individual icons, one for each volume.

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ The data seems to have consistent file naming, so you could script it, or you could use something like wget -m. I'm guessing there is a LOT of data there, so make sure you have the disk space. $\endgroup$ – user21 Nov 22 '19 at 2:37
2
$\begingroup$

You can do it, I will leave you to decide whether you should... As an experiment, I downloaded the mrox_3374 subdirectory of files and the result came out to 3.7 GB. Assuming the other 99 "volumes"/directories are the same, this multiplies out to 370 GB.

I downloaded these data using wget which is a standard utility on Unix/Linux and Mac computers. It is also available for Windows systems; this Wget for Windows 10 tutorial should be of some help in getting it installed and going.

Once this is installed you can do:

wget -np -nH -c --cut-dirs 4 -r https://pds-imaging.jpl.nasa.gov/data/mro/mars_reconnaissance_orbiter/ctx/mrox_3374/

The key flags are -r which means recurse into directories, -np which means don't ascend to the parent directory (to stop it sucking down the entire website), -c which will continue the transfer if it fails and you restart the wget and the --cut-dirs 4 which cuts 4 levels of the directories off the recursive download (the data/mro/mars_reconnaissance_orbiter/ctx parts)

If you really, really want to suck down the whole thing you could do:

wget -np -nH -c --cut-dirs 3 -r https://pds-imaging.jpl.nasa.gov/data/mro/mars_reconnaissance_orbiter/ctx/

and that should grab everything.

$\endgroup$
2
  • $\begingroup$ Success! I needed only one volume. Window 10 ( or may be my work place firewall) created some some hiccups. It would tell in the middle of execution that "wget has stopped working" . I started the program 4 times and thanks to wget robustness each time it started where it had left. (got volume 3448, 1.83GB). $\endgroup$ – qqqqq Nov 22 '19 at 23:18
  • 1
    $\begingroup$ I use these datasets. CTX has a data volume of >10TB, while MARCI is closer to around 1TB (significant rounding involved in these numbers). That is just the real data - the .IMG files. Add to that all the preview/browse data and you need more space. As of PDS release 51, CTX has around 106,000 .IMG files, and MARCI is the same to within 1000-2000. $\endgroup$ – Stuart Robbins Mar 8 '20 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.