7
$\begingroup$

I noticed that most of the tools for cosmological simulation (e.g. large scale structure or single galaxy evolution) are based on GADGET. It only allows to use CPU.

Why not GPU? What is the main obstacle? Is it computationally ineffective for this type of simulations, and if so, why?

Are there any other tools for cosmological simulations that make use of GPU?

Improve this question
$\endgroup$
3
  • $\begingroup$ This doesn't answer your question, but there's lots of good info & links related to large-scale structure sims: astronomy.stackexchange.com/q/55010/16685 $\endgroup$
    – PM 2Ring
    Commented Aug 13 at 9:09
  • 1
    $\begingroup$ I think that although they're faster, GPUs don't offer the same amount of memory, which is an issue for large-scale simulations. There are indeed simulations carried out on GPUs, but I think a more practical issue is that the codes are developed by astrophysicists, not computer scientists, so often are perhaps not the most optimal solutions. $\endgroup$
    – pela
    Commented Aug 13 at 13:31
  • $\begingroup$ There's maybe another issue in that GPUs are often optimized for single-precision (or even half-precision) floating-point operations, and are slower when doing double-precision calculations. $\endgroup$
    – Peter Erwin
    Commented Aug 13 at 14:25

1 Answer 1

Reset to default
8
$\begingroup$

I think there are two ways to answer your question.

The first is to point out that GADGET was developed in the late 1990s and early 2000s, when GPUs were still a bit primitive, programming support for them not very portable, and most supercomputers did not include GPUs.

The second is to note that there are updated versions of GADGET that use GPUs. Ragagnin et al. 2020, "Gadget3 on GPUs with OpenACC", discusses progress being made in rewriting GADGET-3 to take advantage of GPUs (while still being usable on systems without GPUs). I suspect a number of your questions might be answerable by reading it; I'll note some interesting comments about why GPUs are not perfect for cosmological simulations:

Here below we list various limitations that prevent an easy porting of the whole code Gadget3 to the GPUs:

  • The code do not benefit from vectorisation because it stores data in arrays of large data structures (≈ 500B each) that do not fit modern architecture caches. Chang- ing the data layout to a structure of arrays would require a massive refactoring ef- fort and introduce additional memory movement (of packing and unpacking data) in the domain decomposition.
  • The use of blocking MPI communications (to exchange neighbouring particles between MPI ranks) poses a limit in fully utilising GPUs and CPUs. [...]
  • GPUs memories have less capacity than their host memories, thus simulations that keeps all data in GPUs will require more computing nodes than CPU only runs.
  • Gadget3 has been built over a decennial effort of developers who implemented various flavours of gravity, SPH solvers, and sub-resolution models that have been extensively tested; rewriting these modules using CUDA/OpenCL languages would imply a massive rewrite of portions of such modules with associated risks of adding mistakes.

For these reasons, a directive-based approach that uses OpenACC [10] has been adopted. This reduces modifications of the ongoing development of Gadget3 and further- more makes it possible to still run the code on CPU-only systems.

(MPI is used for communication between different nodes.)

As an example of the memory-limitation problem, they note that in a typical simulation

"each node was allocating 4GB for the Barnes Hut tree, 22GB for the basic quantities used in gravity (e.g. position, mass, acceleration ecc..), and additional 14GB for the SPH-only part (that is split in den- sity computation and hydro-force computation), 0.6GB for the metal evolution and an additional amount of 4GB for the active particle list and to store the Hilbert space-filling-curve keys, for a grand total of 40GB per node."

and then go on to point out, "It is clear that a 16GB GPU system (as for instance, the ones in Piz Daint) would not be able to store the same number of particles of its underlying host."

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Peter Erwin to the rescue! $\endgroup$
    – uhoh
    Commented Aug 19 at 9:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .