Reading GADGET data (experimental)

Mera's analysis layer is code-blind, so a reader only has to fill the standard structs. This page adds a frontend for the GADGET HDF5 snapshot format — also written by GIZMO, AREPO, SWIFT, EAGLE and IllustrisTNG — so getvar, projection, msum, center_of_mass and the rest run on its particles unchanged.

Usage

getinfo / getparticles auto-detect GADGET from the HDF5 Header group:

using Mera
info = getinfo(200, "/path/to/gadget/run")   # finds snap…_200.hdf5, simcode = "GADGET"
part = getparticles(info)                      # a PartDataType (:x,:y,:z,:vx,:vy,:vz,:mass,:id,:family)

msum(part); center_of_mass(part); getvar(part, :vx)

:family is the GADGET particle type — 0 gas, 1 halo/DM, 2 disk, 3 bulge, 4 stars, 5 boundary/BH. On a large snapshot, restrict to a subset with the frontend directly to keep RAM bounded:

stars = getparticles_gadget(info; families=[4])      # just the star particles
dm    = getparticles_gadget(info; families=[1])      # just the dark matter

Masses come from each type's Masses dataset, or from Header/MassTable for types that store a single per-type value (e.g. dark matter).

Loading a spatial sub-region

getparticles honours the RAMSES spatial-window arguments xrange/yrange/zrange (with center/range_unit). Particles outside the box are dropped per type as they are read, so a sub-region of a huge snapshot never accumulates in memory:

# the central 20 % box (fractions of the box, relative to its centre)
part = getparticles(info; xrange=[-0.1, 0.1], yrange=[-0.1, 0.1], zrange=[-0.1, 0.1],
                    center=[:bc], range_unit=:standard)

The result equals a full load filtered by getvar(:x), and the window is recorded in part.ranges. Combine with families= (on the frontend) to load, say, only the stars in a region.

Selecting which columns are read

Gas cells carry many fields, and reading all of them is usually the dominant memory cost. vars= limits the load to the ones you need:

gas = getparticles_gadget(info; families=[0], vars=[:rho, :u, :ne])   # + the base columns

The nine base columns (:x, :y, :z, :vx, :vy, :vz, :mass, :id, :family) always load — they define the object. vars selects among the stored gas fields (:rho, :u, :ne, :metallicity, :sfr, :nh, :mach, :gpot, :bx, :by, :bz) plus :volume, which is derived from :rho and pulls it in automatically. Omit vars (or pass :all) for everything present.

Measured on a 16-chunk CAMELS snapshot, same rows each time:

Derived quantities need their inputs: getvar(:T), :p and :cs are computed from :u (and :T uses :ne for the μ correction), so a load without :u raises a clear error naming what is missing rather than failing later. An unknown symbol in vars is rejected immediately, listing the valid ones.

filterdata(gas, Above(:T, 1e5))              # 1e5 in CODE units — probably not what you meant
filterdata(gas, Above(:T, 1e5; unit=:K))     # 1e5 K

Halo membership: the group catalogue

A snapshot says where the gas is; the group catalogue says which halo it belongs to. In the IllustrisTNG workflow membership comes from the halo finder, not from geometry — the reference reader illustris_python has no spatial selection at all, only loadHalo(id). Mera offers the same idiom:

info = getinfo(33, "/path/to/TNG50-4")   # simcode = "AREPO"
gc   = getgroups(info)                    # the FoF catalogue, all chunks
gc.n                                      # number of groups
gc.GroupMassType[1, 1]                    # group 1, gas mass  [1e10 M⊙/h]

gas   = getparticles(info; families=[0], halo=0)   # exactly that group's cells
stars = getparticles(info; families=[4], halo=0)

The catalogue is found beside the snapshot (basePath/groups_NNN/ next to basePath/snapdir_NNN/), so pointing getinfo at the snapshot is enough. A partial catalogue is an error, not a short answer: getgroups compares the groups it read against the header's Ngroups_Total and refuses if they disagree — an incomplete download otherwise yields a perfectly plausible-looking mass function.

Both conventions are checked against the catalogue's own published masses. Recomputing GroupMassType from the particles of TNG50-4 snapshot 33:

i.e. exact to float32 round-off. (Masses are in 1e10 M⊙/h; convert with info.constants.Msol and info.H0/100.)

Mass-conserving maps: pick the pixel to suit the window

A projection deposits each cell over a stencil, so cells near the frame edge put part of that stencil outside the map and the share is not deposited. The loss is a boundary effect and shrinks as the pixel does — on a ±300 kpc window of the TNG halo:

Nothing is wrong at 20 kpc — the missing 2.7 % is genuinely outside the frame — but if you need the map to account for the window's mass, either use a pixel small compared with the window, or project a frame larger than the region you care about and measure inside it. That is the same advice the off-axis page gives for edge pixels, and it applies to every deposit mode.

Multi-file (chunked) snapshots

Large runs split one snapshot across snap_NNN.0.hdf5 … snap_NNN.K.hdf5, usually inside a snapdir_NNN/ directory. Mera resolves the whole set and streams it chunk by chunk, so a windowed load never holds more than one chunk at a time. Nothing extra is required — point getinfo at the run directory, the snapdir_NNN/, or any single chunk (its siblings are gathered automatically):

info = getinfo(24, "/path/to/run")                    # finds snapdir_024/, all 16 chunks
gas  = getparticles_gadget(info; families=[0])        # every chunk, one table

Two properties of real snapshots that this has to get right, and which single-file fixtures cannot exercise:

• A particle type may be absent from chunk 0. Field discovery therefore scans forward until it finds a chunk that carries the type, rather than assuming chunk 0 is representative. (The reference reader illustris_python does the same.) Verified on a CAMELS snapshot whose PartType1/4/5 appear only in later chunks — all load with their full counts.
• Counts above 2³² are split across two header fields. NumPart_Total is combined with NumPart_Total_HighWord, so a snapshot with 9.5 × 10⁹ gas cells reports that rather than the truncated low word.

The total is taken from the header, so for a full-box load the columns are sized once up front instead of grown per chunk.

Worked example: the yt GadgetDiskGalaxy sample

The yt GadgetDiskGalaxy sample is a z ≈ 1.9 galaxy with ~11.9M particles (4.3M gas, 4.8M DM, 2.3M disk, 451k stars). getinfo prints the overview:

julia> info = getinfo(200, "/data/gadget_diskgalaxy/GadgetDiskGalaxy");

Code: GADGET
output: 200  time: 0.34483  redshift: 1.9
boxlen = 64000.0
particles: 4334546 gas, 4786616 halo/DM, 2333848 disk, 450921 stars, 1149 bndry/BH  (total 11907080)
-------------------------------------------------------

and the particles plot directly — the dark-matter cosmic web and the star particles tracing the forming galaxy:

dm = getparticles_gadget(info; families=[1]); st = getparticles_gadget(info; families=[4])
# scatter getvar(dm,:x) vs getvar(dm,:y), and the stars — or project with a finer lmax/res

Gas analysis (AREPO / IllustrisTNG)

For gas (PartType0) the cell fields are read alongside the kinematics, so the full thermodynamic analysis runs through the usual getvar/projection calls — in physical units:

info = getinfo(59, "/data/TNG/halo_59")          # IllustrisTNG cutout (AREPO)
gas  = getparticles(info; families=[0])           # PartType0 → :rho,:u,:ne,:metallicity,:sfr,:volume

getvar(gas, :rho, :g_cm3)                          # physical density (a/h applied for cosmological runs)
getvar(gas, :T, :K)                                # temperature [K] from :u (+ :ne when present)
                                                   # (omit the unit and you get code units)
getvar(gas, :metallicity)                          # mass-fraction metallicity

pdf(gas, :rho); profile(gas, :r_sphere, :T)        # PDFs / radial profiles on the gas

Maps come from the particle projection, which deposits each Voronoi cell at its position. Extensive maps (surface density) are mass-conserving to machine precision (Σ pixel·area == msum); intensive maps (temperature, metallicity) take a weighting:

projection(gas, :sd, :Msol_pc2)                          # surface density (mass-conserving)
projection(gas, :T, weighting=:mass)                     # mass-weighted  ⟨T⟩  (dense gas)
projection(gas, :T, weighting=:volume)                   # volume-weighted ⟨T⟩ (diffuse gas)
projection(gas, :sd, :Msol_pc2, weighting=:sph)          # SPH-kernel: smear each cell over its footprint

(Tip: for a halo cutout in a large box, zoom the projection — center=[…], range_unit=:kpc, xrange=[-R,R], … — and raise res so the window spans enough pixels, since res is full-box-relative.)

Units

GADGET data is in code units (commonly length kpc/h, mass 10¹⁰ M⊙/h, velocity km/s, with h the dimensionless Hubble parameter). The base CGS units are read from the Header (UnitLength/Mass/Velocity_in_*), so getvar(gas, :rho, :g_cm3), getvar(part, :vx, :km_s), etc. return physical quantities out of the box; the unit_length/unit_density/unit_velocity keywords override them. Cosmological metadata (Time = scale factor, Redshift, HubbleParam, Omega0/OmegaLambda) is read from the Header, and a run is treated as cosmological when OmegaLambda > 0. For cosmological runs the comoving→physical factors are applied automatically: positions ∝ a/h, density ∝ h²/a³, mass ∝ 1/h, and velocities carry the √a factor — so a code that returns physical values to getvar needs no manual h/a bookkeeping. (A non-cosmological run, OmegaLambda = 0, is left untouched: a = 1, Time is a physical time.)

How it maps onto Mera's structs

Each PartTypeN group has Coordinates/Velocities (3×N), ParticleIDs, and optionally Masses. The reader concatenates the requested types into one PartDataType with columns (:x,:y,:z, :vx,:vy,:vz, :mass, :id, :family) — positions in code units [0, boxlen], exactly the convention the RAMSES/PLUTO particle readers use, so the particle analysis works unchanged. The mapping is verified data-free in test/60_gadget_reader_tests.jl (a synthesised GADGET file with a MassTable fallback) and on the real GadgetDiskGalaxy sample.

Reference readers

The GADGET HDF5 layout is shared and well-documented; this frontend agrees with the origin tools:

• The GADGET snapshot specification — the Header + PartType* group format (Coordinates/ Velocities/Masses/ParticleIDs, Header/MassTable/BoxSize/…), documented in the GADGET-4 guide and reused by GIZMO, AREPO, SWIFT, EAGLE and IllustrisTNG.
• yt — its particle frontends read the same format and select sub-volumes lazily via data objects; Mera's load-time window mirrors that on the particle list. The GadgetDiskGalaxy sample used above comes from the yt sample-data collection.

See also

• Multi-code support — the code-blind architecture and the other readers.
• getparticles, getvar, projection — the particle analysis that runs on GADGET data.