MERA Files API Reference

savedata(data; fmode, path="./", kwargs...)
savedata(data, fmode::Symbol; path="./", kwargs...)
savedata(data, path::String; fmode, kwargs...)
savedata(data, path::String, fmode::Symbol; kwargs...)

Save a Mera data object to a compressed JLD2 file, a "Mera file", which reloads much faster than the original RAMSES output.

One file per output holds any number of data types (hydro, particles, gravity, clumps, RT, sinks). The file is named after the output number, e.g. output_00300.jld2, and keeps the table together with the simulation information and the options it was loaded with.

Arguments

• data: the object to save, from gethydro, getparticles, getgravity, getclumps, getrt or getsinks.
• fmode::Symbol: what to do with the file. Nothing is written unless it is given, so a file is never overwritten by accident.
  • :write: create the file, or replace an existing one.
  • :append: add a further data type to an existing file. A data type that is already in the file is not replaced; that raises an error.
• path::String = "./": the folder to write to.

Keywords

• fname = "output_": the start of the file name; the five-digit output number follows.
• compress: LZ4 compression is on by default; compress=false switches it off. Older compressor objects are accepted, and Zlib or Bzip2 fall back to LZ4 with a warning.
• comments: a text stored with the data, e.g. a description of the simulation.
• dataformat::Symbol = :JLD2: the only format.
• merafile_version::Float64 = 1.0: the file layout version; there is only one.
• sidecar::Bool = true: also write output_XXXXX.jld2.meta.json, a readable record of the data types in the file, their provenance and the environment; an append adds to it. The global switch sidecar overrides this keyword.
• anonymize::Bool = false: in the sidecar, keep only the last two parts of the simulation path.
• note::String = "": a free text stored in the sidecar.
• include_script::Bool = false: also store the full text of the running script (or the code cells of the notebook) in the sidecar. Its name and SHA-256 are always recorded, when Julia knows the script.
• verbose::Bool = true: print what is written.

Examples

julia> savedata(gas, :write; path="mera_files/");     # new file output_00300.jld2

julia> savedata(part, :append; path="mera_files/");   # add the particles to it

julia> viewdata(300; path="mera_files/")              # what the file holds

See also loaddata, viewdata, infodata, convertdata.

Returns nothing.

loaddata(output::Int, datatype::Symbol; path="./", kwargs...)
loaddata(output::Int, path::String, datatype::Symbol; kwargs...)
loaddata(output::Int; datatype, path="./", kwargs...)
loaddata(output::Int, path::String; datatype, kwargs...)

Load one data type from a Mera file written by savedata.

The stored table is read in full; a region given by the ranges is then cut out. Files written by any earlier Mera version can be read.

Arguments

• output::Int: the output number in the file name, e.g. 300 for output_00300.jld2.
• datatype::Symbol: the data type to load: :hydro, :particles, :gravity, :clumps, :rt or :sinks. It is required, as a positional argument or as a keyword.
• path::String = "./": the folder of the file.

Keywords

Selection:

• xrange, yrange, zrange: [min, max] of the region to keep, in range_unit and relative to center. missing (the default) means the full extent.
• center = [0, 0, 0]: the origin of the ranges, in range_unit. [:bc] is the box centre.
• range_unit::Symbol = :standard: the unit of the ranges and the centre. :standard means fractions of the box (0 to 1); any length unit in info.scale works, e.g. :kpc.

File:

• fname = "output_": the start of the file name, as given to savedata.
• regenerate_scales::Bool = true: recompute the physical constants and unit scales with the current Mera version. false keeps exactly what was stored.

Output:

• verbose::Bool = true: print what is read.
• myargs::ArgumentsType: set the ranges, center, range_unit and verbose in one object. A keyword given directly takes precedence.

Returns

The stored object, e.g. a HydroDataType for :hydro, the same as the function that first read it returned.

Examples

julia> gas = loaddata(300, :hydro; path="mera_files/");

julia> part = loaddata(300, "mera_files/", :particles; xrange=[-10, 10], yrange=[-10, 10],
                       zrange=[-2, 2], center=[:bc], range_unit=:kpc);

See also savedata, viewdata, infodata.

viewdata(output::Int, path::String="./"; fname="output_", showfull=false, verbose=true) -> Dict

Show what a Mera file written by savedata contains, without loading the data.

For each stored data type it reports the Mera and file versions, the compression, the size on disk and in memory, and the conversion statistics when the file was made by convertdata.

Arguments

• output::Int: the output number in the file name, e.g. 300 for output_00300.jld2.
• path::String = "./": the folder of the file; also a keyword.

Keywords

• fname = "output_": the start of the file name, as given to savedata.
• showfull::Bool = false: also print the full tree of the file.
• verbose::Bool = true: print the overview.

Returns

A Dict with one entry per stored data type (e.g. "hydro", "particles") and "FileSize".

Examples

julia> viewdata(300, "mera_files/");

julia> keys(viewdata(300, "mera_files/"; verbose=false))

See also savedata, loaddata, infodata.

Get the simulation overview from RAMSES, saved in JLD2 == function getinfo

infodata(output::Int;
         path::String="./",
         fname = "output_",
         datatype::Any=:nothing,
         verbose::Bool=true)

return InfoType

Keyword Arguments

• output: timestep number
• path: the path to the output JLD2 file relative to the current folder or absolute path
• fname: "output"-> filename = "output***.jld2" by default, can be changed to "myname***.jld2"
• verbose:: informations are printed on the screen by default

Examples

# read simulation information from output `1` in current folder
julia> info = infodata(1) # filename="output_00001.jld2"

# read simulation information from output `420` in given folder (relative path to the current working folder)
julia> info = infodata(420, path="../MySimFolder/")

# or simply use
julia> info = infodata(420, "../MySimFolder/")



# get an overview of the returned field-names
julia> propertynames(info)

# a more detailed overview
julia> viewfields(info)
...
julia> viewallfields(info)
...
julia> namelist(info)
...
julia> makefile(info)
...
julia> timerfile(info)
...
julia> patchfile(info)
...

exportdata(result, file; format=:auto, vars=nothing, source=nothing,
           length_unit=:standard, sidecar=true, anonymize=false, note="",
           include_script=false, verbose=true) -> String

Write a Mera result to a file that other tools read, and a sidecar <file>.meta.json that records what it holds and where it came from.

Use it to publish the data behind a figure, or to continue in Python, IDL or a spreadsheet. The file holds plain arrays; the sidecar holds the provenance, the units and the environment that ran (the Reproducibility page explains it). To keep a result for Mera itself, use savemap or savedata instead: they round-trip the full object.

Arguments

• result: what to write.
  • a projection from projection: every map, plus the pixel centres x and y;
  • a profile, phase or pdf result;
  • a timeseries table;
  • a movie from getmovie: the frames as a cube, plus outputs and times;
  • an array, a number, or a NamedTuple of arrays, e.g. getvar columns. Pass source so the sidecar knows which snapshot they came from.
• file::String: the file to write. Its extension selects the format.

Keywords

• format::Symbol = :auto: the format, :jld2, :hdf5, :csv, :npy, :npz, :json or :fits; :auto takes it from the extension.
• vars = nothing: write only these arrays, e.g. vars=:sd or vars=[:mean, :median]. The coordinates (pixel centres, bin edges) are always kept.
• source = nothing: the data object or InfoType the result was computed from. Needed for arrays, numbers and timeseries tables, which do not carry it; a map, profile, phase or pdf carries it already.
• length_unit::Symbol = :standard: the unit of a map's pixel centres, extent and pixel size, e.g. :kpc. :standard keeps code units. Maps only.
• sidecar::Bool = true: write <file>.meta.json. The global switch sidecar overrides it.
• anonymize::Bool = false: in the sidecar, keep only the last two parts of the simulation path. The host name is never recorded.
• note::String = "": a free text stored in the sidecar, e.g. which figure this is.
• include_script::Bool = false: also store the full text of the running script (or the code cells of the notebook) in the sidecar. Its name and SHA-256 are always recorded, when Julia knows the script.
• verbose::Bool = true: print what was written.

Python and other C-order readers see the axes of an HDF5 or JLD2 array reversed (a Julia nx × ny map has shape (ny, nx) in h5py). NumPy files are written in Fortran order, so there a[i, j] is A[i+1, j+1]. The sidecar states the layout of each file.

Returns

The path of the data file, as a String.

Examples

p = projection(gas, [:sd, :T], [:Msol_pc2, :K])
exportdata(p, "fig3.h5"; length_unit=:kpc)          # + fig3.h5.meta.json
exportdata(p, "fig3_sd.npy"; vars=:sd)              # one map for numpy
exportdata(p, "fig3.csv"; length_unit=:kpc)         # x, y, sd, T per pixel

prof = profile(gas, :r_cylinder, :rho; xunit=:kpc, unit=:nH)
exportdata(prof, "fig4.csv"; note="Figure 4, radial density")

exportdata((rho=getvar(gas, :rho, :nH), T=getvar(gas, :T, :K)), "cells.npz"; source=gas)

using FITSIO
exportdata(p, "fig3.fits"; length_unit=:kpc)

See also savemap, savedata, provenance, export_vtk.

writesidecar(file, source=nothing; note="", anonymize=false, include_script=false, verbose=true)

Write the sidecar <file>.meta.json for a file that Mera did not write itself, typically a figure saved with Makie, PyPlot or Plots. The sidecar is the same as for exportdata: the SHA-256 of the file, the provenance of the snapshot, the input files of the run, the analysis environment and script, and the Julia and Mera that ran.

Call it right after the save:

save("fig3.png", fig)               # Makie
writesidecar("fig3.png", p)         # p: the projection shown in the figure

Arguments

• file::String: an existing file.
• source: the Mera result the file shows (a projection, profile, data object, movie or InfoType), or a vector of them for a figure with several panels. With nothing the sidecar still records the file, the analysis and the environment, but no snapshot.

Keywords

• note::String = "": a free text stored in the sidecar, e.g. "Figure 3 of the paper".
• anonymize::Bool = false: keep only the last two parts of the simulation path.
• include_script::Bool = false: also store the full text of the running script (or the code cells of the notebook). Its name and SHA-256 are always recorded, when Julia knows the script.
• verbose::Bool = true: print the sidecar's path.

Returns

The sidecar's path as a String, or nothing when sidecars are switched off with sidecar(false) or the sidecar could not be written (a warning says why). An existing sidecar of the file is replaced.

Examples

writesidecar("fig3.png", p; note="Figure 3")
writesidecar("evolution.pdf", [p10, p20, p30])   # one panel per snapshot
readsidecar("fig3.png")                          # read it back

See also readsidecar, sidecaroverview, exportdata.

readsidecar(file; verify=true) -> SidecarRecord

Read the sidecar of a file written by savedata, savemap, savemovie, save_clumps, exportdata or writesidecar, and print a short summary in the REPL.

Arguments

• file::String: the data file (fig3.h5) or its sidecar (fig3.h5.meta.json).

Keywords

• verify::Bool = true: compare the data file with the SHA-256 in the sidecar, so status says whether it changed since it was written. Reading a large Mera file takes a few seconds per GB; verify=false skips it.

Returns

A SidecarRecord; r.data holds the whole record, e.g. r.data["source"]["output"].

Examples

r = readsidecar("fig3.h5")          # prints what fig3.h5 holds and where it came from
r.status                            # :unchanged, :changed or :missing
r.data["environment"]["julia_version"]

See also sidecaroverview, writesidecar, exportdata, sidecar.

sidecaroverview(dir="."; recursive=false, verify=false, verbose=true) -> Vector{SidecarRecord}

List every sidecar (*.meta.json) in a folder as a table: the file, what it holds, the snapshot it came from, when it was written, and whether the data file is still there.

Arguments

• dir::String = ".": the folder to look in.

Keywords

• recursive::Bool = false: also look in subfolders.
• verify::Bool = false: compare every data file with its recorded SHA-256 (slow for large files); the status column then says unchanged or changed.
• verbose::Bool = true: print the table.

Returns

A Vector of SidecarRecord, sorted by file name.

Examples

sidecaroverview("results/")                  # one line per saved or exported file
recs = sidecaroverview("results/"; verify=true, verbose=false)
filter(r -> r.status === :changed, recs)     # files edited after they were written

See also readsidecar, exportdata.

bundle(dir="."; title=nothing, description=nothing, creators=nothing, license=nothing,
       keywords=String[], simulation=nothing, recursive=true, verify=true, verbose=true) -> String

Describe a finished analysis folder in one record, ready to publish, for example on Zenodo. bundle reads the sidecars in the folder and writes three files into it:

• ro-crate-metadata.json: an RO-Crate that lists every file with its size, type and SHA-256, and for each file written by Mera what made it: the Mera version, the script and the snapshot, from its sidecar;
• software.bib: the BibTeX from citation;
• simulation.json: the simulationrecord, when simulation is given.

Nothing is copied, so the folder needs no extra disk space. Running it again replaces the three files.

Arguments

• dir::String = ".": the analysis folder.

Keywords

• title, description, creators, license, keywords: who made the analysis and under which license, as for datacite: creators as "Family, Given" or (name=…, orcid=…, affiliation=…), license as an SPDX identifier such as "CC-BY-4.0". Missing ones are read from an analysis.toml in the folder (the same fields as simulation.toml, see meraproject); the title falls back to the folder name.
• simulation = nothing: the simulation the analysis used: a folder, an InfoType or a SimulationRecord. Its record is written to simulation.json.
• recursive::Bool = true: include subfolders. Hidden files and folders are left out.
• verify::Bool = true: check each file against the SHA-256 in its sidecar, and warn about files that changed after they were written.
• verbose::Bool = true: print a summary.

Returns

The path of ro-crate-metadata.json.

Examples

bundle("galaxy_paper"; simulation="/data/mw_L10",
       creators=["Behrendt, Manuel"], license="CC-BY-4.0")

See also meraproject, sidecaroverview, citation, datacite.

meraproject(dir; title=nothing, verbose=true) -> String

Start a new analysis folder with its own Julia environment, so every sidecar records exactly the packages that ran. It writes:

• Project.toml, with Mera at the version that runs now;
• analysis.jl, a first script;
• analysis.toml, the title, authors and license of the analysis, for bundle;
• README.md, the few steps from the first run to publishing.

Nothing is installed and no network is needed: the first julia --project=. in the folder, followed by ] instantiate, installs Mera there.

Arguments

• dir::String: the new folder. It may exist, but must not hold a Project.toml yet.

Keywords

• title::String = nothing: the title of the analysis, stored in analysis.toml; defaults to the folder name.
• verbose::Bool = true: print the next steps.

Returns

The folder's path.

Examples

meraproject("galaxy_paper")
# then, in a terminal:
#   cd galaxy_paper
#   julia --project=.
#   ] instantiate

See also bundle, writesidecar.

SidecarRecord

What readsidecar and sidecaroverview return for one sidecar: file (the data file), sidecar (the .meta.json), data (the parsed JSON, index it like a dictionary, e.g. r.data["source"]["output"]) and status: :unchanged or :changed when the file's SHA-256 was checked, :missing when the data file is gone, :unchecked otherwise.

SimulationRecord

What simulationrecord returns: data, the record as an ordered dictionary (index it directly, e.g. rec["resolution"]["levelmax"]), and path, the simulation folder. It prints a short summary in the REPL. Write it as JSON with writesimulationrecord, or turn it into DataCite metadata with datacite.

Converts full simulation data into a compressed/uncompressed JLD2 format:

This function provides a comprehensive data conversion workflow for RAMSES simulation data, converting multiple data types into compressed JLD2 format with full benchmarking and threading control capabilities.

Features:

• Multi-datatype support: Handles :hydro, :particles, :gravity, :clumps, :rt, :sinks data types
• Threading control: Configurable threading for performance optimization (excluding clumps)
• Compression options: Multiple compression algorithms with automatic selection
• Spatial filtering: Select specific data ranges with flexible unit support
• Benchmarking: Comprehensive timing and performance statistics storage
• Progress tracking: Optional progress bars and verbose output modes
• Memory management: Automatic memory cleanup and usage tracking

Data Processing Workflow:

1. Configuration: Parse arguments and setup threading/compression parameters
2. Data Loading: Sequential loading of requested datatypes with optional threading
3. Data Writing: Compressed storage to JLD2 format with timing measurements
4. Statistics: Comprehensive benchmark and threading information storage

function convertdata(output::Int; datatypes::Array{<:Any,1}=[missing], path::String="./", fpath::String="./", fname = "output", compress::Any=nothing, comments::Any=nothing, lmax::Union{Int, Missing}=missing, xrange::Array{<:Any,1}=[missing, missing], yrange::Array{<:Any,1}=[missing, missing], zrange::Array{<:Any,1}=[missing, missing], center::CenterType=[0., 0., 0.], rangeunit::Symbol=:standard, smallr::Real=0., smallc::Real=0., verbose::Bool=true, showprogress::Bool=true, maxthreads::Int=Threads.nthreads(), myargs::ArgumentsType=ArgumentsType() )

return statistics_dictionary

Arguments

Required:

• output: RAMSES output number to convert

Optional Keywords:

• datatypes: Array of datatypes to convert.
  • Default: [missing] → converts all available data (:hydro, :gravity, :particles, :clumps, :rt, :sinks)
  • Examples: [:hydro, :particles], [:hydro], :particles
• path: Path to RAMSES simulation folders (default: "./").
• fpath: Output path for JLD2 files (default: "./")
• fname: Base filename for output files (default: "output_")
  • Final filename: fname + output_number + ".jld2"
• compress: Compression settings
  • Default: nothing → automatic LZ4 compression
  • Options: false (no compression), LZ4FrameCompressor(), Bzip2Compressor(), ZlibCompressor()
  • Requires: CodecZlib, CodecBzip2, or CodecLz4 packages for specific compressors
• comments: String description of the simulation (stored in file metadata)
• lmax: Maximum AMR level to process
  • Default: missing → uses all available levels
  • Limits data processing to specified refinement level
• xrange, yrange, zrange: Spatial selection ranges [min, max]
  • Default: [missing, missing] → full simulation box
  • Units specified by range_unit, relative to center
  • Zero-length ranges (min=max=0) converted to maximum extent
• center: Coordinate center for spatial ranges
  • Default: [0., 0., 0.] → simulation origin
  • Special values: [:bc], [:boxcenter] → box center
  • Mixed: [value, :bc, :bc] → custom center per axis
• range_unit: Units for spatial ranges and center
  • Options: :standard (code units), :Mpc, :kpc, :pc, :mpc, :ly, :au, :km, :cm
  • See viewfields(info.scale) for available unit conversions
• smallr: Lower density threshold (0 = inactive)
• smallc: Lower thermal pressure threshold (0 = inactive)
• max_threads: Threading control for data loading operations
  • Default: Threads.nthreads() → uses all available Julia threads
  • Applied to: :hydro, :gravity, :particles data loading
  • NOT applied to: :clumps and :sinks data (single-threaded)
  • Examples: max_threads=4 (limit to 4 threads), max_threads=1 (single-threaded)
• verbose: Enable detailed console output (default: true)
• show_progress: Enable progress bars during data loading (default: true)
• myargs: Pre-configured ArgumentsType struct to override multiple parameters

Return Value:

Returns a comprehensive statistics dictionary containing:

• TimerOutputs: Detailed timing for reading/writing operations
• threading: Threading configuration and system information
• benchmark: Performance metrics, compression ratios, processing times
• viewdata: Metadata about the converted dataset
• size: Memory usage and file size information

Method Overloads:

• convertdata(output::Int64; ...) → Full parameter interface
• convertdata(output::Int64, datatypes::Vector{Symbol}; ...) → Direct datatype specification
• convertdata(output::Int64, datatypes::Symbol; ...) → Single datatype conversion

Threading Behavior:

• Threaded operations: File-level parallelism for :hydro, :gravity, :particles
• Single-threaded: :clumps and :sinks data processing (threading not beneficial)
• Thread safety: All operations use thread-safe file I/O and memory management
• Performance: Optimal threading automatically balances CPU files across threads

Usage Examples:

Convert all datatypes with default settings and available threads

stats = convertdata(42, path, fpath)

Convert specific datatypes with threading control

stats = convertdata(42, [:hydro, :particles], path="source/folder", fpath="export/folder", max_threads=4)

Spatial selection with different compression

stats = convertdata(42, xrange=[-10, 10], yrange=[-10, 10], rangeunit=:Mpc, compress=Bzip2Compressor(), maxthreads=8)

Access performance statistics println("Total time: ", stats["benchmark"]["totalprocessingtimeseconds"]) println("Compression ratio: ", stats["benchmark"]["compressionratio"]) println("Threads used: ", stats["threading"]["effective_threads"])

batch_convert_mera(input_dir::String, output_dir::String, 
                           start_output::Int, end_output::Int;
                           requested_threads::Int=Threads.nthreads(),
                           safety_margin::Float64=DEFAULT_SAFETY_MARGIN,
                           min_threads::Int=DEFAULT_MIN_THREADS,
                           max_threads::Int=DEFAULT_MAX_THREADS,
                           skip_existing::Bool=true,
                           show_confirmation::Bool=true,
                           compress=nothing) -> Dict

Main function for safe multithreaded batch conversion with active safety margin monitoring.

This function coordinates the entire conversion process including:

1. System resource validation and safety checks
2. File discovery and filtering by output number range
3. Thread count optimization based on system constraints
4. User confirmation and information display
5. Multithreaded conversion with real-time monitoring
6. Comprehensive results reporting and recommendations

Parameter Details

Required Parameters

• input_dir: Source directory containing old JLD2 files with version issues
• output_dir: Destination directory for converted files (created if doesn't exist)
• start_output: Starting output number for conversion range (inclusive)
• end_output: Ending output number for conversion range (inclusive)

Performance Tuning Parameters

• requested_threads: Desired number of conversion threads (default: all available)
• safety_margin: Memory usage threshold as decimal 0.0-1.0 (default: 0.8 = 80%)
• min_threads: Minimum thread count even under resource constraints (default: 1)
• max_threads: Maximum thread count regardless of system capacity (default: 64)

Behavior Control Parameters

• skip_existing: Skip files that already exist in output directory (default: true)
• show_confirmation: Display user confirmation prompt before starting (default: true)
• compress: Compression codec for the output files, matching savedata's API (default: nothing → LZ4FrameCompressor()). Pass false to write uncompressed files, or a specific codec instance (LZ4FrameCompressor(), ZlibCompressor(), Bzip2Compressor()) for finer control.
• sidecar: also write <file>.meta.json next to each converted file, as savedata does: the data types it holds, the provenance of the snapshot, the environment, and the file it was converted from (default: true). The global switch sidecar overrides it.

Safety Margin System

The safety_margin parameter is now actively used throughout the process:

Pre-Conversion Phase

• Validates current system memory usage
• Adjusts thread recommendations based on available memory within safety limits
• Warns user if current usage already exceeds margin

During Conversion Phase

• Monitors memory usage before each file load operation
• Checks memory after data loading (peak usage point)
• Triggers automatic garbage collection on violations
• Counts total violations for reporting

Post-Conversion Phase

• Reports final memory state and violation statistics
• Provides recommendations for future conversions based on violation patterns

Return Value

Returns comprehensive dictionary with conversion statistics:

• success: Number of files successfully converted
• failed: Number of files that failed conversion
• skipped: Number of files skipped (already existed)
• safety_violations: Number of times memory exceeded safety margin
• conversion_time: Total time spent in conversion (seconds)
• threads_used: Actual number of threads used
• final_memory_usage_percent: Memory usage percentage at completion

Error Handling Strategy

The function handles errors gracefully:

• Individual file failures don't stop the batch
• Out-of-memory errors receive specific guidance
• System resource violations trigger automatic recovery
• All errors are logged with specific context

Example Usage

Basic conversion with default safety settings: results = batchconvertmera("/data/old", "/data/new", 100, 200)

Conservative conversion for large files: results = batchconvertmera("/data/old", "/data/new", 100, 200; requestedthreads=4, safetymargin=0.9)

High-performance conversion with monitoring: results = batchconvertmera("/data/old", "/data/new", 100, 200; requestedthreads=16, safetymargin=0.7, skip_existing=false)

interactive_mera_converter(input_dir::String, output_dir::String;
                                   safety_margin::Float64=DEFAULT_SAFETY_MARGIN,
                                   min_threads::Int=DEFAULT_MIN_THREADS,
                                   max_threads::Int=DEFAULT_MAX_THREADS)

Interactive mode for file conversion with comprehensive user guidance and system information.

This function provides a user-friendly interface that:

1. Displays comprehensive system information and constraints
2. Analyzes available files and detects potential issues
3. Guides user through range and thread count selection
4. Provides intelligent recommendations based on system state
5. Executes conversion with all safety monitoring features

User Experience Flow

System Information Display

• Shows CPU core count and memory configuration
• Displays current memory usage and safety margin status
• Indicates thread count limits and recommendations
• Warns about any current resource constraints

File Analysis and Validation

• Scans input directory for valid RAMSES files
• Reports total file count and available output ranges
• Detects and reports gaps in file sequences
• Helps user identify potential data integrity issues

Guided Parameter Selection

• Prompts for output number range with sensible defaults
• Recommends thread count based on system capacity and safety constraints
• Allows user override with explanation of implications
• Provides real-time feedback on selections

Safety-Monitored Execution

• Calls main conversion function with user-selected parameters
• Provides same comprehensive monitoring as batch function
• Returns complete results for user review

Parameters

• input_dir: Source directory containing old JLD2 files
• output_dir: Destination directory for converted files
• safety_margin: Memory usage threshold (default: 0.8 = 80%)
• min_threads: Minimum thread count (default: 1)
• max_threads: Maximum thread count (default: 64)

Example Usage

Basic interactive mode interactivemeraconverter("/data/old", "/data/new")

Conservative interactive mode for large files interactivemeraconverter("/data/old", "/data/new"; safetymargin=0.9, maxthreads=8)

optimize_mera_io(simulation_path::String, output_num::Int; benchmark=false, quiet=false)

Automatically optimize Mera I/O settings based on your simulation characteristics.

This is the easiest way to get optimal performance - just provide your simulation path and output number, and Mera will analyze your data and apply the best settings.

Arguments

• simulation_path: Path to your RAMSES simulation directory
• output_num: Output number to analyze (e.g., 300)
• benchmark=false: Set to true to run performance benchmarks for fine-tuning
• quiet=false: Set to true to suppress output messages

Returns

• true if optimization was successful, false otherwise

Examples

# Basic automatic optimization
optimize_mera_io("/Volumes/Storage/Simulations/mw_L10", 300)

# With benchmarking for maximum performance
optimize_mera_io("/Volumes/Storage/Simulations/mw_L10", 300, benchmark=true)

# Quiet mode for scripts
optimize_mera_io("/path/to/sim", 300, quiet=true)

What it does

1. Analyzes your simulation (file count, sizes, AMR structure)
2. Recommends optimal buffer size based on simulation characteristics
3. Enables file metadata caching for faster repeat operations
4. Optionally benchmarks different settings to find the absolute best performance

Simulation size recommendations

• Small (< 50 files): 32KB buffer
• Medium (50-200 files): 64KB buffer
• Large (200-500 files): 128KB buffer
• Very large (500-1000 files): 256KB buffer
• Huge (> 1000 files): 512KB buffer

configure_mera_io(; buffer_size="auto", cache=true, large_buffers=true, show_config=true)

Manually configure Mera I/O settings with user-friendly parameters.

Arguments

• buffer_size: Buffer size as string ("32KB", "64KB", "128KB", "256KB", "512KB") or "auto"
• cache=true: Enable file metadata caching for faster repeat operations
• large_buffers=true: Enable large buffer optimizations
• show_config=true: Display the applied configuration

Examples

# Use 128KB buffer with caching
configure_mera_io(buffer_size="128KB")

# Disable caching
configure_mera_io(buffer_size="64KB", cache=false)

# Maximum performance for very large simulations
configure_mera_io(buffer_size="512KB", cache=true, large_buffers=true)

# Minimal settings for small simulations
configure_mera_io(buffer_size="32KB", large_buffers=false)

Buffer size recommendations

• "32KB": Small simulations (< 50 CPU files)
• "64KB": Medium simulations (50-200 CPU files) - Default
• "128KB": Large simulations (200-500 CPU files)
• "256KB": Very large simulations (500-1000 CPU files)
• "512KB": Huge simulations (> 1000 CPU files)

Returns true.

show_mera_config()

Display current Mera I/O configuration settings.

Shows buffer size, caching status, and performance-related settings in a user-friendly format.

Example

julia> show_mera_config()
🔧 MERA I/O CONFIGURATION
========================
Buffer size:     128KB (131072 bytes)
File caching:    Enabled ✅
Large buffers:   Enabled ✅
Cache entries:   3 files cached
Status:          Optimized for large simulations

Returns nothing.

reset_mera_io()

Reset Mera I/O settings to default values.

This clears any custom buffer sizes, disables optimizations, and clears the cache. Useful if you want to start fresh or if you're experiencing issues.

Example

julia> reset_mera_io()
🔄 MERA I/O RESET
=================
✅ Buffer size reset to 64KB (default)
✅ File caching enabled (default)
✅ Cache cleared (0 entries removed)
✅ Settings reset to defaults

Returns nothing.

mera_io_status()

Quick status check of Mera I/O configuration and performance.

Returns a summary of current settings and cache performance in a compact format.

Example

julia> mera_io_status()
"I/O: 128KB buffer, cache enabled (5 files), optimized ✅"

benchmark_mera_io(simulation_path::String, output_num::Int; 
                 test_sizes=["32KB", "64KB", "128KB", "256KB"])

Benchmark different I/O configurations to find optimal settings for your specific simulation.

This function tests various buffer sizes with your actual data to determine which configuration gives the best performance on your system.

Arguments

• simulation_path: Path to your RAMSES simulation directory
• output_num: Output number to test with
• test_sizes: Array of buffer sizes to test (as strings)

Returns

• Dictionary with benchmark results and recommended optimal settings

Example

# Standard benchmark
results = benchmark_mera_io("/path/to/simulation", 300)

# Custom buffer sizes to test
results = benchmark_mera_io("/path/to/simulation", 300, 
                           test_sizes=["64KB", "128KB", "256KB", "512KB"])

# Access results
optimal_buffer = results["optimal_buffer_size"]
performance_gain = results["performance_improvement"]

What it does

1. Tests each buffer size with your actual simulation data
2. Measures getinfo() and gethydro() performance
3. Identifies the optimal buffer size for your system
4. Automatically applies the best settings
5. Returns detailed performance comparison

smart_io_setup(simulation_path::String, output_num::Int; benchmark=false, verbose=true)

Intelligent I/O setup that combines analysis and optional benchmarking.

Returns true when the automatic analysis worked and the settings were applied, false otherwise.

configure_adaptive_io(simulation_path::String, output_num::Int; verbose=true)

Automatically configure I/O settings based on simulation characteristics.

Returns true when the settings were applied, false when the simulation could not be analysed and the defaults are kept.

ensure_optimal_io!(info::InfoType; force_reoptimize=false, verbose=false)

Automatically ensures optimal I/O settings based on simulation characteristics. This function is called transparently by gethydro(), getparticles(), and getgravity().

Arguments

• info: InfoType object from getinfo()
• force_reoptimize=false: Force re-optimization even if already optimized
• verbose=false: Enable detailed output (usually disabled for transparent operation)

Returns

• true if optimization was applied/verified, false if failed

reset_auto_optimization!()

Reset the automatic optimization state, forcing re-optimization on next data load.

Returns nothing.

show_auto_optimization_status()

Display the current status of automatic I/O optimization.

Returns nothing.

benchmark_buffer_sizes(simulation_path::String, output_num::Int; 
                      test_sizes=[32768, 65536, 131072, 262144], verbose=true)

Benchmark different buffer sizes to find the optimal setting for this specific simulation.

Returns a Dict with "optimal_buffer", "optimal_time" and "all_results", or nothing when no buffer size could be timed.

get_simulation_characteristics(simulation_path::String, output_num::Int)

Analyze simulation folder to determine optimal I/O settings. Returns a dictionary with simulation characteristics and recommended settings.

show_mera_cache_stats()

Print what the simulation-metadata cache currently holds: the number of entries and the path behind each one.

Use it to check whether a getinfo call was served from cache before reaching for clear_mera_cache!.

Returns nothing.

clear_mera_cache!()

Empty the cache of simulation metadata that Mera keeps between getinfo calls, and report how many entries were dropped.

Useful when a simulation folder changed on disk during a session and you want the next getinfo to re-read it rather than reuse what it saw earlier. See show_mera_cache_stats to inspect the cache first.

Returns nothing.