GitHub

Export/Import Data (ASCII/Binary)

Run it yourself

This page is also an executable Jupyter notebook — open / download ExportImportData.ipynb. The notebooks run end-to-end and double as part of Mera's test suite.

This notebook presents several ways to export your data.

Used libraries in this tutorial:

  • DelimitedFiles, Serialization (comes with Julia)
  • IndexedTables, FileIO, CSVFiles, JLD, JLD2, CodecZlib, HDF5, Numpy, FITS, Matlap, GZip (needs to be installed)

Load The Data

import Pkg; Pkg.activate("../."); Pkg.build("Mera")
  Activating project at `/Volumes/FASTStorage/Simulations/Notebooks/Mera-Docs/version_1.1`
using Mera
*__   __ _______ ______   _______ 
|  |_|  |       |    _ | |   _   |
|       |    ___|   | || |  |_|  |
|       |   |___|   |_||_|       |
|       |    ___|    __  |       |
| ||_|| |   |___|   |  | |   _   |
|_|   |_|_______|___|  |_|__| |__|
Mera v1.8.0
# Example-data root. Point this at your own simulation folder, or set the
# MERA_EXAMPLES environment variable; every path below is built from it.
MERA_EXAMPLES = get(ENV, "MERA_EXAMPLES", "/Volumes/FASTStorage/Simulations/Mera-Tests");

info = getinfo(400, "$MERA_EXAMPLES/RAMSES/manu_sim_sf_L14", verbose=false)
hydro = gethydro(info, :rho, smallr=1e-5, lmax=10)
particles = getparticles(info, :mass);
[Mera]: Get hydro data: 2026-08-06T10:36:44.702

Key vars=(:level, :cx, :cy, :cz)
Using var(s)=(1,) = (:rho,) 

domain:
xmin::xmax: 0.0 :: 1.0  	==> 0.0 [kpc] :: 48.0 [kpc]
ymin::ymax: 0.0 :: 1.0  	==> 0.0 [kpc] :: 48.0 [kpc]
zmin::zmax: 0.0 :: 1.0  	==> 0.0 [kpc] :: 48.0 [kpc]

📊 Processing Configuration:
   Total CPU files available: 2048
   Files to be processed: 2048
   Compute threads: 4
   GC threads: 4


✓ File processing complete! Combining results...
✓ Data combination complete!
Final data size: 4879946 cells, 1 variables
Creating Table from 4879946 cells with max 4 threads...
  Threading: 4 threads for 5 columns
  Max threads requested: 4
  Available threads: 4
  Using parallel processing with 4 threads
  Creating IndexedTable with 5 columns...
✓ Table created in 0.979 seconds
Memory used for data table :186.1557970046997 MB
-------------------------------------------------------

[Mera]: Get particle data: 2026-08-06T10:37:14.140

Using threaded processing with 4 threads
Key vars=(:level, :x, :y, :z, :id)
Using var(s)=(4,) = (:mass,) 

domain:
xmin::xmax: 0.0 :: 1.0  	==> 0.0 [kpc] :: 48.0 [kpc]
ymin::ymax: 0.0 :: 1.0  	==> 0.0 [kpc] :: 48.0 [kpc]
zmin::zmax: 0.0 :: 1.0  	==> 0.0 [kpc] :: 48.0 [kpc]

Processing 2048 CPU files using 4 threads
Mode: Threaded processing
Combining results from 4 thread(s)...
Found 5.089390e+05 particles
Memory used for data table :19.415205001831055 MB
-------------------------------------------------------
println("Cells: ", length(hydro.data))
println("Particles: ", length(particles.data))
Cells: 4879946
Particles: 508939

Define a function to preview the first lines of the created ASCII files:

function viewheader(filename, lines)
    open(filename) do f
        line = 1
        while line<=lines
            x = readline(f)
            println(x)
            line += 1
        end
    end
end
viewheader (generic function with 1 method)

Collect The Data For Export

# Get the cell and particle positions relative to the box-center
# Choose the relevant units
# The function getvar returns a dictionary containing a 1d-array for each quantity
hvals = getvar(hydro, [:x,:y,:z,:cellsize,:rho], [:kpc,:kpc,:kpc,:kpc,:g_cm3], center=[:boxcenter]);
pvals = getvar(hydro, [:x,:y,:z,:mass], [:kpc,:kpc,:kpc,:Msol], center=[:boxcenter]);
hvals
Dict{Any, Any} with 5 entries:
  :cellsize => [0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75  …  …
  :y        => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, …
  :rho      => [6.76838e-28, 6.76838e-28, 6.76838e-28, 6.76838e-28, 6.76838e-28…
  :z        => [-23.625, -22.875, -22.125, -21.375, -20.625, -19.875, -19.125, …
  :x        => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, …
pvals
Dict{Any, Any} with 4 entries:
  :y    => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.…
  :z    => [-23.625, -22.875, -22.125, -21.375, -20.625, -19.875, -19.125, -18.…
  :mass => [4217.58, 4217.58, 4217.58, 4217.58, 4217.58, 4217.58, 4217.58, 4217…
  :x    => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.…

ASCII: DelimitedFiles Library

using DelimitedFiles

Save into an ASCII file with no header, comma separated:

open("simulation_hydro.csv", "w") do io
    writedlm(io, [hvals[:x] hvals[:y] hvals[:z] hvals[:cellsize] hvals[:rho]], ",")
end

Check the first lines in the file:

viewheader("simulation_hydro.csv", 5)
-23.625000000015312,-23.625000000015312,-23.625000000015312,0.7500000000004861,6.768382184513761e-28
-23.625000000015312,-23.625000000015312,-22.875000000014825,0.7500000000004861,6.768382184513761e-28
-23.625000000015312,-23.625000000015312,-22.12500000001434,0.7500000000004861,6.768382184513761e-28
-23.625000000015312,-23.625000000015312,-21.375000000013856,0.7500000000004861,6.768382184513761e-28
-23.625000000015312,-23.625000000015312,-20.62500000001337,0.7500000000004861,6.768382184513761e-28

Use a different syntax; save into file with header and tab-separated values:

header = ["x/kpc" "y/kpc" "z/kpc" "cellsize/kpc" "rho/g_cm3"]
valsrray = [hvals[:x] hvals[:y] hvals[:z] hvals[:cellsize] hvals[:rho]] # Array with the columns
writedlm("simulation_hydro.dat", [header ; valsrray], "\t")
viewheader("simulation_hydro.dat", 5)
x/kpc	y/kpc	z/kpc	cellsize/kpc	rho/g_cm3
-23.625000000015312	-23.625000000015312	-23.625000000015312	0.7500000000004861	6.768382184513761e-28
-23.625000000015312	-23.625000000015312	-22.875000000014825	0.7500000000004861	6.768382184513761e-28
-23.625000000015312	-23.625000000015312	-22.12500000001434	0.7500000000004861	6.768382184513761e-28
-23.625000000015312	-23.625000000015312	-21.375000000013856	0.7500000000004861	6.768382184513761e-28

Write the particles data into an ASCII file with header:

header = ["x/kpc" "y/kpc" "z/kpc" "mass/Msol"]
valsrray = [pvals[:x] pvals[:y] pvals[:z] pvals[:mass]]
writedlm("simulation_particles.dat", [header ; valsrray], "\t")
viewheader("simulation_particles.dat", 5)
x/kpc	y/kpc	z/kpc	mass/Msol
-23.625000000015312	-23.625000000015312	-23.625000000015312	4217.583427040147
-23.625000000015312	-23.625000000015312	-22.875000000014825	4217.583427040147
-23.625000000015312	-23.625000000015312	-22.12500000001434	4217.583427040147
-23.625000000015312	-23.625000000015312	-21.375000000013856	4217.583427040147

ASCII: Save IndexedTables Database into a CSV-File with FileIO

using FileIO

See for documentation https://github.com/JuliaIO/FileIO.jl/tree/master/docs

The simulation data is stored in a IndexedTables database:

particles.data
Table with 508939 rows, 6 columns:
level  x           y        z        id      mass
───────────────────────────────────────────────────────
6      0.00462947  22.3885  24.571   327957  1.13606e-5
6      0.109066    22.3782  21.5844  116193  1.13606e-5
6      0.238211    28.7537  24.8191  194252  1.13606e-5
6      0.271366    22.7512  31.5681  130805  1.13606e-5
6      0.312574    16.2385  23.7591  162174  1.13606e-5
6      0.314957    28.2084  30.966   320052  1.13606e-5
6      0.328337    4.59858  23.5001  292889  1.13606e-5
6      0.420712    27.6688  26.5735  102940  1.13606e-5
6      0.509144    33.1737  23.9789  183902  1.13606e-5
6      0.565516    25.9409  26.0579  342278  1.13606e-5
6      0.587289    9.60231  23.8477  280020  1.13606e-5
6      0.592878    25.5519  21.3079  64182   1.13606e-5
⋮
14     37.6271     25.857   23.8833  437164  1.13606e-5
14     37.6299     25.8403  23.9383  421177  1.13606e-5
14     37.6301     25.8502  23.9361  478941  1.13606e-5
14     37.6326     25.8544  23.9383  428429  1.13606e-5
14     37.6528     25.8898  23.9928  467148  1.13606e-5
14     37.6643     25.9061  23.9945  496129  1.13606e-5
14     37.6813     25.8743  23.9789  435636  1.13606e-5
14     37.7207     25.8623  23.8775  476398  1.13606e-5
14     38.173      25.8862  23.7978  347919  1.13606e-5
14     38.1738     25.8914  23.7979  403094  1.13606e-5
14     38.1739     25.8905  23.7992  381503  1.13606e-5
FileIO.save("database_partilces.csv", particles.data)
viewheader("database_partilces.csv", 5)
"level","x","y","z","id","mass"
6,0.004629472789625229,22.388543919075275,24.571021484979347,327957,1.1360607549574087e-5
6,0.1090659052277639,22.3782196217294,21.58442789512976,116193,1.1360607549574087e-5
6,0.2382109772356709,28.753723953405462,24.81911909925676,194252,1.1360607549574087e-5
6,0.271365638325332,22.751224267806695,31.568145104287826,130805,1.1360607549574087e-5

Export selected variables from the datatable:

using Mera.IndexedTables

See for documentation https://juliacomputing.github.io/JuliaDB.jl/latest/

FileIO.save("database_partilces.csv", select(particles.data, (:x,:y,:mass)) )
viewheader("database_partilces.csv", 5)
"x","y","mass"
0.004629472789625229,22.388543919075275,1.1360607549574087e-5
0.1090659052277639,22.3782196217294,1.1360607549574087e-5
0.2382109772356709,28.753723953405462,1.1360607549574087e-5
0.271365638325332,22.751224267806695,1.1360607549574087e-5

Binary: Save Multiple Data into a JLD File

See for documentation: https://github.com/JuliaIO/JLD.jl

using JLD
jldopen("mydata.jld", "w") do file
    write(file, "hydro", hvals ) 
    write(file, "particles", pvals ) 
end

Open file for read and get an overview of the stored dataset:

file = jldopen("mydata.jld","r")
Julia data file version 0.1.3: mydata.jld
names(file)
2-element Vector{String}:
 "hydro"
 "particles"
hydrodata = read(file, "hydro")
Dict{Any, Any} with 5 entries:
  :x        => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, …
  :y        => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, …
  :rho      => [6.76838e-28, 6.76838e-28, 6.76838e-28, 6.76838e-28, 6.76838e-28…
  :z        => [-23.625, -22.875, -22.125, -21.375, -20.625, -19.875, -19.125, …
  :cellsize => [0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75  …  …
particledata = read(file, "particles")
Dict{Any, Any} with 4 entries:
  :y    => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.…
  :z    => [-23.625, -22.875, -22.125, -21.375, -20.625, -19.875, -19.125, -18.…
  :mass => [4217.58, 4217.58, 4217.58, 4217.58, 4217.58, 4217.58, 4217.58, 4217…
  :x    => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.…

Compare stored with original data:

hydrodata == hvals
true
particledata == pvals
true

Binary: Compress Data into a gz-File

using CodecZlib, Serialization

See for documentation: https://github.com/JuliaIO/CodecZlib.jl

fo= GzipCompressorStream( open("sample-data.jls.gz", "w") ); serialize(fo, hvals); close(fo)
hydrodata1 = deserialize( GzipDecompressorStream( open("sample-data.jls.gz", "r") ) )
Dict{Any, Any} with 5 entries:
  :x        => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, …
  :y        => [-23.625, -23.625, -23.625, -23.625, -23.625, -23.625, -23.625, …
  :rho      => [6.76838e-28, 6.76838e-28, 6.76838e-28, 6.76838e-28, 6.76838e-28…
  :z        => [-23.625, -22.875, -22.125, -21.375, -20.625, -19.875, -19.125, …
  :cellsize => [0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 0.75  …  …
hydrodata1 == hvals
true

Prepare variable-array:

varsarray = [hvals[:x] hvals[:y] hvals[:z] hvals[:cellsize] hvals[:rho]]
4879946×5 Matrix{Float64}:
 -23.625   -23.625    -23.625      0.75      6.76838e-28
 -23.625   -23.625    -22.875      0.75      6.76838e-28
 -23.625   -23.625    -22.125      0.75      6.76838e-28
 -23.625   -23.625    -21.375      0.75      6.76838e-28
 -23.625   -23.625    -20.625      0.75      6.76838e-28
 -23.625   -23.625    -19.875      0.75      6.76838e-28
 -23.625   -23.625    -19.125      0.75      6.76838e-28
 -23.625   -23.625    -18.375      0.75      6.76838e-28
 -23.625   -23.625    -17.625      0.75      6.76838e-28
 -23.625   -23.625    -16.875      0.75      6.76838e-28
 -23.625   -23.625    -16.125      0.75      6.76838e-28
 -23.625   -23.625    -15.375      0.75      6.76838e-28
 -23.625   -23.625    -14.625      0.75      6.76838e-28
   ⋮                                         
  14.6953    1.94531   -0.0703125  0.046875  3.59298e-26
  14.6953    1.94531   -0.0234375  0.046875  3.80161e-26
  14.6953    1.94531    0.0234375  0.046875  4.29495e-26
  14.6953    1.94531    0.0703125  0.046875  3.96562e-26
  14.6953    1.99219   -0.164063   0.046875  2.49252e-26
  14.6953    1.99219   -0.117188   0.046875  2.58237e-26
  14.6953    1.99219   -0.0703125  0.046875  2.71999e-26
  14.6953    1.99219   -0.0234375  0.046875  2.79827e-26
  14.6953    2.03906   -0.164063   0.046875  2.39398e-26
  14.6953    2.03906   -0.117188   0.046875  2.44115e-26
  14.6953    2.03906   -0.0703125  0.046875  2.57262e-26
  14.6953    2.03906   -0.0234375  0.046875  2.61481e-26
fo= GzipCompressorStream( open("sample-data2.jls.gz", "w") ); serialize(fo, varsarray); close(fo)

Read the data again:

hydrodata2 = deserialize( GzipDecompressorStream( open("sample-data2.jls.gz", "r") ) )
4879946×5 Matrix{Float64}:
 -23.625   -23.625    -23.625      0.75      6.76838e-28
 -23.625   -23.625    -22.875      0.75      6.76838e-28
 -23.625   -23.625    -22.125      0.75      6.76838e-28
 -23.625   -23.625    -21.375      0.75      6.76838e-28
 -23.625   -23.625    -20.625      0.75      6.76838e-28
 -23.625   -23.625    -19.875      0.75      6.76838e-28
 -23.625   -23.625    -19.125      0.75      6.76838e-28
 -23.625   -23.625    -18.375      0.75      6.76838e-28
 -23.625   -23.625    -17.625      0.75      6.76838e-28
 -23.625   -23.625    -16.875      0.75      6.76838e-28
 -23.625   -23.625    -16.125      0.75      6.76838e-28
 -23.625   -23.625    -15.375      0.75      6.76838e-28
 -23.625   -23.625    -14.625      0.75      6.76838e-28
   ⋮                                         
  14.6953    1.94531   -0.0703125  0.046875  3.59298e-26
  14.6953    1.94531   -0.0234375  0.046875  3.80161e-26
  14.6953    1.94531    0.0234375  0.046875  4.29495e-26
  14.6953    1.94531    0.0703125  0.046875  3.96562e-26
  14.6953    1.99219   -0.164063   0.046875  2.49252e-26
  14.6953    1.99219   -0.117188   0.046875  2.58237e-26
  14.6953    1.99219   -0.0703125  0.046875  2.71999e-26
  14.6953    1.99219   -0.0234375  0.046875  2.79827e-26
  14.6953    2.03906   -0.164063   0.046875  2.39398e-26
  14.6953    2.03906   -0.117188   0.046875  2.44115e-26
  14.6953    2.03906   -0.0703125  0.046875  2.57262e-26
  14.6953    2.03906   -0.0234375  0.046875  2.61481e-26

Compare original with loaded data:

hydrodata2 == varsarray
true

Store array with header:

header = ["x/kpc" "y/kpc" "z/kpc" "cellsize/kpc" "rho/g_cm3"]
fo= GzipCompressorStream( open("sample-data3.jls.gz", "w") ); serialize(fo, [header ; varsarray]); close(fo)
hydrodata3 = deserialize( GzipDecompressorStream( open("sample-data3.jls.gz", "r") ) )
4879947×5 Matrix{Any}:
    "x/kpc"     "y/kpc"     "z/kpc"    "cellsize/kpc"   "rho/g_cm3"
 -23.625     -23.625     -23.625      0.75             6.76838e-28
 -23.625     -23.625     -22.875      0.75             6.76838e-28
 -23.625     -23.625     -22.125      0.75             6.76838e-28
 -23.625     -23.625     -21.375      0.75             6.76838e-28
 -23.625     -23.625     -20.625      0.75             6.76838e-28
 -23.625     -23.625     -19.875      0.75             6.76838e-28
 -23.625     -23.625     -19.125      0.75             6.76838e-28
 -23.625     -23.625     -18.375      0.75             6.76838e-28
 -23.625     -23.625     -17.625      0.75             6.76838e-28
 -23.625     -23.625     -16.875      0.75             6.76838e-28
 -23.625     -23.625     -16.125      0.75             6.76838e-28
 -23.625     -23.625     -15.375      0.75             6.76838e-28
   ⋮                                                   
  14.6953      1.94531    -0.0703125  0.046875         3.59298e-26
  14.6953      1.94531    -0.0234375  0.046875         3.80161e-26
  14.6953      1.94531     0.0234375  0.046875         4.29495e-26
  14.6953      1.94531     0.0703125  0.046875         3.96562e-26
  14.6953      1.99219    -0.164063   0.046875         2.49252e-26
  14.6953      1.99219    -0.117188   0.046875         2.58237e-26
  14.6953      1.99219    -0.0703125  0.046875         2.71999e-26
  14.6953      1.99219    -0.0234375  0.046875         2.79827e-26
  14.6953      2.03906    -0.164063   0.046875         2.39398e-26
  14.6953      2.03906    -0.117188   0.046875         2.44115e-26
  14.6953      2.03906    -0.0703125  0.046875         2.57262e-26
  14.6953      2.03906    -0.0234375  0.046875         2.61481e-26

Other File Formats

  • JLD2 https://github.com/JuliaIO/JLD2.jl
  • HDF5 https://github.com/JuliaIO/HDF5.jl
  • Numpy https://github.com/fhs/NPZ.jl
  • FITS https://github.com/JuliaAstro/FITSIO.jl
  • FITS https://github.com/emmt/EasyFITS.jl
  • Matlab https://github.com/JuliaIO/MAT.jl
  • GZip https://github.com/JuliaIO/GZip.jl