MERA Files API Reference

Save loaded simulation data into a compressed/uncompressed JLD2 format:

• write new file; add datatype to existing file
• running number is taken from original RAMSES folders
• use different compression methods
• add a string to describe the simulation
• toggle verbose mode

function savedata( dataobject::DataSetType;
                    path::String="./",
                    fname = "output_",
                    fmode::Any=nothing,
                    dataformat::Symbol=:JLD2,
                    compress::Any=nothing,
                    comments::Any=nothing,
                    merafile_version::Float64=1.,
                    verbose::Bool=true)
return 

Arguments

Required:

• dataobject: needs to be of type: "HydroDataType", "PartDataType", "GravDataType", "ClumpDataType", "RtDataType"
• fmode: nothing is written/appended by default to avoid overwriting files by accident. Need: fmode=:write (new file, or overwrite an existing file); fmode=:append adds a further datatype to an existing file. Re-appending a datatype that is already stored in the file is not supported and raises an error (existing datatypes are not overwritten in place).

Predefined/Optional Keywords:

• path: path to save the file; default is local path.
• fname: default name of the files "output_" and the running number is added. Change the string to apply a user-defined name.
• dataformat: currently, only JLD2 can be selected.
• compress: by default LZ4 compression is activated. compress=false deactivates it.

This build (JLD2 0.6) compresses with LZ4 (best ratio); a legacy LZ4FrameCompressor() is accepted, and ZlibCompressor()/Bzip2Compressor() fall back to LZ4 with a warning.

• comments: add a string that includes e.g. a description about your simulation
• merafile_version: default: 1.; current only version
• verbose: print timestamp and further information on screen; default: true

Defined Methods - function defined for different arguments

• savedata( dataobject::DataSetType; ...) # note: fmode needs to be given for action!
• savedata( dataobject::DataSetType, fmode::Symbol; ...)
• savedata( dataobject::DataSetType, path::String; ...)
• savedata( dataobject::DataSetType, path::String, fmode::Symbol; ...)

Read stored simulation data into a dataobject:

• supported datatypes: HydroDataType, PartDataType, GravDataType, ClumpDataType
• select a certain data range (data is fully loaded; the selected subregion is returned)
• toggle verbose mode

function loaddata(output::Int; path::String="./",
            fname = "output_",
            datatype::Symbol,
            xrange::Array{<:Any,1}=[missing, missing],
            yrange::Array{<:Any,1}=[missing, missing],
            zrange::Array{<:Any,1}=[missing, missing],
            center::Array{<:Any,1}=[0., 0., 0.],
            range_unit::Symbol=:standard,
            verbose::Bool=true,
            myargs::ArgumentsType=ArgumentsType() )

return dataobject

Arguments

Required:

• output: output number
• datatype: :hydro, :particles, :gravity, :clumps or :rt

Predefined/Optional Keywords:

• path: path to the file; default is local path.
• fname: default name of the files "output_" and the running number is added. Change the string to apply a user-defined name.
• xrange: the range between [xmin, xmax] in units given by argument range_unit and relative to the given center; zero length for xmin=xmax=0. is converted to maximum possible length
• yrange: the range between [ymin, ymax] in units given by argument range_unit and relative to the given center; zero length for ymin=ymax=0. is converted to maximum possible length
• zrange: the range between [zmin, zmax] in units given by argument range_unit and relative to the given center; zero length for zmin=zmax=0. is converted to maximum possible length
• range_unit: the units of the given ranges: :standard (code units), :Mpc, :kpc, :pc, :mpc, :ly, :au , :km, :cm (of typye Symbol) ..etc. ; see for defined length-scales viewfields(info.scale)
• myargs: pass a struct of ArgumentsType to pass several arguments at once and to overwrite default values of xrange, yrange, zrange, center, range_unit, verbose
• verbose: print timestamp and further information on screen; default: true

Defined Methods - function defined for different arguments

• loaddata(output::Int64; ...) # opens first datatype in the file
• loaddata(output::Int64, datatype::Symbol; ...)
• loaddata(output::Int64, path::String; ...)
• loaddata(output::Int64, path::String, datatype::Symbol; ...)

Get overview of stored datatypes:

• compression
• versions of the used/loaded compression
• MERA/MERA-file version
• compressed/uncompressed data size
• returns stored conversion statistics, when available (created by convertdata-function)

function viewdata(output::Int;
        path::String="./",
        fname = "output_",
        showfull::Bool=false,
        verbose::Bool=true)

return overview (dictionary)

Arguments

Required:

• output: output number
• datatype: :hydro, :particles, :gravity, :clumps or :rt

Predefined/Optional Keywords:

• path: the path to the output JLD2 file relative to the current folder or absolute path
• fname: "output"-> filename = "output***.jld2" by default, can be changed to "myname***.jld2"
• showfull: shows the full data tree of the datafile
• verbose:: informations are printed on the screen by default

Get the simulation overview from RAMSES, saved in JLD2 == function getinfo

infodata(output::Int;
         path::String="./",
         fname = "output_",
         datatype::Any=:nothing,
         verbose::Bool=true)

return InfoType

Keyword Arguments

• output: timestep number
• path: the path to the output JLD2 file relative to the current folder or absolute path
• fname: "output"-> filename = "output***.jld2" by default, can be changed to "myname***.jld2"
• verbose:: informations are printed on the screen by default

Examples

# read simulation information from output `1` in current folder
julia> info = infodata(1) # filename="output_00001.jld2"

# read simulation information from output `420` in given folder (relative path to the current working folder)
julia> info = infodata(420, path="../MySimFolder/")

# or simply use
julia> info = infodata(420, "../MySimFolder/")



# get an overview of the returned field-names
julia> propertynames(info)

# a more detailed overview
julia> viewfields(info)
...
julia> viewallfields(info)
...
julia> namelist(info)
...
julia> makefile(info)
...
julia> timerfile(info)
...
julia> patchfile(info)
...

Converts full simulation data into a compressed/uncompressed JLD2 format:

This function provides a comprehensive data conversion workflow for RAMSES simulation data, converting multiple data types into compressed JLD2 format with full benchmarking and threading control capabilities.

Features:

• Multi-datatype support: Handles :hydro, :particles, :gravity, :clumps data types
• Threading control: Configurable threading for performance optimization (excluding clumps)
• Compression options: Multiple compression algorithms with automatic selection
• Spatial filtering: Select specific data ranges with flexible unit support
• Benchmarking: Comprehensive timing and performance statistics storage
• Progress tracking: Optional progress bars and verbose output modes
• Memory management: Automatic memory cleanup and usage tracking

Data Processing Workflow:

1. Configuration: Parse arguments and setup threading/compression parameters
2. Data Loading: Sequential loading of requested datatypes with optional threading
3. Data Writing: Compressed storage to JLD2 format with timing measurements
4. Statistics: Comprehensive benchmark and threading information storage

function convertdata(output::Int; datatypes::Array{<:Any,1}=[missing], path::String="./", fpath::String="./", fname = "output", compress::Any=nothing, comments::Any=nothing, lmax::Union{Int, Missing}=missing, xrange::Array{<:Any,1}=[missing, missing], yrange::Array{<:Any,1}=[missing, missing], zrange::Array{<:Any,1}=[missing, missing], center::Array{<:Any,1}=[0., 0., 0.], rangeunit::Symbol=:standard, smallr::Real=0., smallc::Real=0., verbose::Bool=true, showprogress::Bool=true, maxthreads::Int=Threads.nthreads(), myargs::ArgumentsType=ArgumentsType() )

return statistics_dictionary

Arguments

Required:

• output: RAMSES output number to convert

Optional Keywords:

• datatypes: Array of datatypes to convert.
  • Default: [missing] → converts all available data (:hydro, :gravity, :particles, :clumps)
  • Examples: [:hydro, :particles], [:hydro], :particles
• path: Path to RAMSES simulation folders (default: "./").
• fpath: Output path for JLD2 files (default: "./")
• fname: Base filename for output files (default: "output_")
  • Final filename: fname + output_number + ".jld2"
• compress: Compression settings
  • Default: nothing → automatic LZ4 compression
  • Options: false (no compression), LZ4FrameCompressor(), Bzip2Compressor(), ZlibCompressor()
  • Requires: CodecZlib, CodecBzip2, or CodecLz4 packages for specific compressors
• comments: String description of the simulation (stored in file metadata)
• lmax: Maximum AMR level to process
  • Default: missing → uses all available levels
  • Limits data processing to specified refinement level
• xrange, yrange, zrange: Spatial selection ranges [min, max]
  • Default: [missing, missing] → full simulation box
  • Units specified by range_unit, relative to center
  • Zero-length ranges (min=max=0) converted to maximum extent
• center: Coordinate center for spatial ranges
  • Default: [0., 0., 0.] → simulation origin
  • Special values: [:bc], [:boxcenter] → box center
  • Mixed: [value, :bc, :bc] → custom center per axis
• range_unit: Units for spatial ranges and center
  • Options: :standard (code units), :Mpc, :kpc, :pc, :mpc, :ly, :au, :km, :cm
  • See viewfields(info.scale) for available unit conversions
• smallr: Lower density threshold (0 = inactive)
• smallc: Lower thermal pressure threshold (0 = inactive)
• max_threads: Threading control for data loading operations
  • Default: Threads.nthreads() → uses all available Julia threads
  • Applied to: :hydro, :gravity, :particles data loading
  • NOT applied to: :clumps data (single-threaded)
  • Examples: max_threads=4 (limit to 4 threads), max_threads=1 (single-threaded)
• verbose: Enable detailed console output (default: true)
• show_progress: Enable progress bars during data loading (default: true)
• myargs: Pre-configured ArgumentsType struct to override multiple parameters

Return Value:

Returns a comprehensive statistics dictionary containing:

• TimerOutputs: Detailed timing for reading/writing operations
• threading: Threading configuration and system information
• benchmark: Performance metrics, compression ratios, processing times
• viewdata: Metadata about the converted dataset
• size: Memory usage and file size information

Method Overloads:

• convertdata(output::Int64; ...) → Full parameter interface
• convertdata(output::Int64, datatypes::Vector{Symbol}; ...) → Direct datatype specification
• convertdata(output::Int64, datatypes::Symbol; ...) → Single datatype conversion

Threading Behavior:

• Threaded operations: File-level parallelism for :hydro, :gravity, :particles
• Single-threaded: :clumps data processing (threading not beneficial)
• Thread safety: All operations use thread-safe file I/O and memory management
• Performance: Optimal threading automatically balances CPU files across threads

Usage Examples:

Convert all datatypes with default settings and available threads

stats = convertdata(42, path, fpath)

Convert specific datatypes with threading control

stats = convertdata(42, [:hydro, :particles], path="source/folder", fpath="export/folder", max_threads=4)

Spatial selection with different compression

stats = convertdata(42, xrange=[-10, 10], yrange=[-10, 10], rangeunit=:Mpc, compress=Bzip2Compressor(), maxthreads=8)

Access performance statistics println("Total time: ", stats["benchmark"]["totalprocessingtimeseconds"]) println("Compression ratio: ", stats["benchmark"]["compressionratio"]) println("Threads used: ", stats["threading"]["effective_threads"])

batch_convert_mera(input_dir::String, output_dir::String, 
                           start_output::Int, end_output::Int;
                           requested_threads::Int=Threads.nthreads(),
                           safety_margin::Float64=DEFAULT_SAFETY_MARGIN,
                           min_threads::Int=DEFAULT_MIN_THREADS,
                           max_threads::Int=DEFAULT_MAX_THREADS,
                           skip_existing::Bool=true,
                           show_confirmation::Bool=true,
                           compress=nothing) -> Dict

Main function for safe multithreaded batch conversion with active safety margin monitoring.

This function coordinates the entire conversion process including:

1. System resource validation and safety checks
2. File discovery and filtering by output number range
3. Thread count optimization based on system constraints
4. User confirmation and information display
5. Multithreaded conversion with real-time monitoring
6. Comprehensive results reporting and recommendations

Parameter Details

Required Parameters

• input_dir: Source directory containing old JLD2 files with version issues
• output_dir: Destination directory for converted files (created if doesn't exist)
• start_output: Starting output number for conversion range (inclusive)
• end_output: Ending output number for conversion range (inclusive)

Performance Tuning Parameters

• requested_threads: Desired number of conversion threads (default: all available)
• safety_margin: Memory usage threshold as decimal 0.0-1.0 (default: 0.8 = 80%)
• min_threads: Minimum thread count even under resource constraints (default: 1)
• max_threads: Maximum thread count regardless of system capacity (default: 64)

Behavior Control Parameters

• skip_existing: Skip files that already exist in output directory (default: true)
• show_confirmation: Display user confirmation prompt before starting (default: true)
• compress: Compression codec for the output files, matching savedata's API (default: nothing → LZ4FrameCompressor()). Pass false to write uncompressed files, or a specific codec instance (LZ4FrameCompressor(), ZlibCompressor(), Bzip2Compressor()) for finer control.

Safety Margin System

The safety_margin parameter is now actively used throughout the process:

Pre-Conversion Phase

• Validates current system memory usage
• Adjusts thread recommendations based on available memory within safety limits
• Warns user if current usage already exceeds margin

During Conversion Phase

• Monitors memory usage before each file load operation
• Checks memory after data loading (peak usage point)
• Triggers automatic garbage collection on violations
• Counts total violations for reporting

Post-Conversion Phase

• Reports final memory state and violation statistics
• Provides recommendations for future conversions based on violation patterns

Return Value

Returns comprehensive dictionary with conversion statistics:

• success: Number of files successfully converted
• failed: Number of files that failed conversion
• skipped: Number of files skipped (already existed)
• safety_violations: Number of times memory exceeded safety margin
• conversion_time: Total time spent in conversion (seconds)
• threads_used: Actual number of threads used
• final_memory_usage_percent: Memory usage percentage at completion

Error Handling Strategy

The function handles errors gracefully:

• Individual file failures don't stop the batch
• Out-of-memory errors receive specific guidance
• System resource violations trigger automatic recovery
• All errors are logged with specific context

Example Usage

Basic conversion with default safety settings: results = batchconvertmera("/data/old", "/data/new", 100, 200)

Conservative conversion for large files: results = batchconvertmera("/data/old", "/data/new", 100, 200; requestedthreads=4, safetymargin=0.9)

High-performance conversion with monitoring: results = batchconvertmera("/data/old", "/data/new", 100, 200; requestedthreads=16, safetymargin=0.7, skip_existing=false)

interactive_mera_converter(input_dir::String, output_dir::String;
                                   safety_margin::Float64=DEFAULT_SAFETY_MARGIN,
                                   min_threads::Int=DEFAULT_MIN_THREADS,
                                   max_threads::Int=DEFAULT_MAX_THREADS)

Interactive mode for file conversion with comprehensive user guidance and system information.

This function provides a user-friendly interface that:

1. Displays comprehensive system information and constraints
2. Analyzes available files and detects potential issues
3. Guides user through range and thread count selection
4. Provides intelligent recommendations based on system state
5. Executes conversion with all safety monitoring features

User Experience Flow

System Information Display

• Shows CPU core count and memory configuration
• Displays current memory usage and safety margin status
• Indicates thread count limits and recommendations
• Warns about any current resource constraints

File Analysis and Validation

• Scans input directory for valid RAMSES files
• Reports total file count and available output ranges
• Detects and reports gaps in file sequences
• Helps user identify potential data integrity issues

Guided Parameter Selection

• Prompts for output number range with sensible defaults
• Recommends thread count based on system capacity and safety constraints
• Allows user override with explanation of implications
• Provides real-time feedback on selections

Safety-Monitored Execution

• Calls main conversion function with user-selected parameters
• Provides same comprehensive monitoring as batch function
• Returns complete results for user review

Parameters

• input_dir: Source directory containing old JLD2 files
• output_dir: Destination directory for converted files
• safety_margin: Memory usage threshold (default: 0.8 = 80%)
• min_threads: Minimum thread count (default: 1)
• max_threads: Maximum thread count (default: 64)

Example Usage

Basic interactive mode interactivemeraconverter("/data/old", "/data/new")

Conservative interactive mode for large files interactivemeraconverter("/data/old", "/data/new"; safetymargin=0.9, maxthreads=8)

optimize_mera_io(simulation_path::String, output_num::Int; benchmark=false, quiet=false)

Automatically optimize Mera I/O settings based on your simulation characteristics.

This is the easiest way to get optimal performance - just provide your simulation path and output number, and Mera will analyze your data and apply the best settings.

Arguments

• simulation_path: Path to your RAMSES simulation directory
• output_num: Output number to analyze (e.g., 300)
• benchmark=false: Set to true to run performance benchmarks for fine-tuning
• quiet=false: Set to true to suppress output messages

Returns

• true if optimization was successful, false otherwise

Examples

# Basic automatic optimization
optimize_mera_io("/Volumes/Storage/Simulations/mw_L10", 300)

# With benchmarking for maximum performance
optimize_mera_io("/Volumes/Storage/Simulations/mw_L10", 300, benchmark=true)

# Quiet mode for scripts
optimize_mera_io("/path/to/sim", 300, quiet=true)

What it does

1. Analyzes your simulation (file count, sizes, AMR structure)
2. Recommends optimal buffer size based on simulation characteristics
3. Enables file metadata caching for faster repeat operations
4. Optionally benchmarks different settings to find the absolute best performance

Simulation size recommendations

• Small (< 50 files): 32KB buffer
• Medium (50-200 files): 64KB buffer
• Large (200-500 files): 128KB buffer
• Very large (500-1000 files): 256KB buffer
• Huge (> 1000 files): 512KB buffer

configure_mera_io(; buffer_size="auto", cache=true, large_buffers=true, show_config=true)

Manually configure Mera I/O settings with user-friendly parameters.

Arguments

• buffer_size: Buffer size as string ("32KB", "64KB", "128KB", "256KB", "512KB") or "auto"
• cache=true: Enable file metadata caching for faster repeat operations
• large_buffers=true: Enable large buffer optimizations
• show_config=true: Display the applied configuration

Examples

# Use 128KB buffer with caching
configure_mera_io(buffer_size="128KB")

# Disable caching
configure_mera_io(buffer_size="64KB", cache=false)

# Maximum performance for very large simulations
configure_mera_io(buffer_size="512KB", cache=true, large_buffers=true)

# Minimal settings for small simulations
configure_mera_io(buffer_size="32KB", large_buffers=false)

Buffer size recommendations

• "32KB": Small simulations (< 50 CPU files)
• "64KB": Medium simulations (50-200 CPU files) - Default
• "128KB": Large simulations (200-500 CPU files)
• "256KB": Very large simulations (500-1000 CPU files)
• "512KB": Huge simulations (> 1000 CPU files)

show_mera_config()

Display current Mera I/O configuration settings.

Shows buffer size, caching status, and performance-related settings in a user-friendly format.

Example

julia> show_mera_config()
🔧 MERA I/O CONFIGURATION
========================
Buffer size:     128KB (131072 bytes)
File caching:    Enabled ✅
Large buffers:   Enabled ✅
Cache entries:   3 files cached
Status:          Optimized for large simulations

reset_mera_io()

Reset Mera I/O settings to default values.

This clears any custom buffer sizes, disables optimizations, and clears the cache. Useful if you want to start fresh or if you're experiencing issues.

Example

julia> reset_mera_io()
🔄 MERA I/O RESET
=================
✅ Buffer size reset to 64KB (default)
✅ File caching enabled (default)
✅ Cache cleared (0 entries removed)
✅ Settings reset to defaults

mera_io_status()

Quick status check of Mera I/O configuration and performance.

Returns a summary of current settings and cache performance in a compact format.

Example

julia> mera_io_status()
"I/O: 128KB buffer, cache enabled (5 files), optimized ✅"

benchmark_mera_io(simulation_path::String, output_num::Int; 
                 test_sizes=["32KB", "64KB", "128KB", "256KB"])

Benchmark different I/O configurations to find optimal settings for your specific simulation.

This function tests various buffer sizes with your actual data to determine which configuration gives the best performance on your system.

Arguments

• simulation_path: Path to your RAMSES simulation directory
• output_num: Output number to test with
• test_sizes: Array of buffer sizes to test (as strings)

Returns

• Dictionary with benchmark results and recommended optimal settings

Example

# Standard benchmark
results = benchmark_mera_io("/path/to/simulation", 300)

# Custom buffer sizes to test
results = benchmark_mera_io("/path/to/simulation", 300, 
                           test_sizes=["64KB", "128KB", "256KB", "512KB"])

# Access results
optimal_buffer = results["optimal_buffer_size"]
performance_gain = results["performance_improvement"]

What it does

1. Tests each buffer size with your actual simulation data
2. Measures getinfo() and gethydro() performance
3. Identifies the optimal buffer size for your system
4. Automatically applies the best settings
5. Returns detailed performance comparison

smart_io_setup(simulation_path::String, output_num::Int; benchmark=false, verbose=true)

Intelligent I/O setup that combines analysis and optional benchmarking.

configure_adaptive_io(simulation_path::String, output_num::Int; verbose=true)

Automatically configure I/O settings based on simulation characteristics.

ensure_optimal_io!(info::InfoType; force_reoptimize=false, verbose=false)

Automatically ensures optimal I/O settings based on simulation characteristics. This function is called transparently by gethydro(), getparticles(), and getgravity().

Arguments

• info: InfoType object from getinfo()
• force_reoptimize=false: Force re-optimization even if already optimized
• verbose=false: Enable detailed output (usually disabled for transparent operation)

Returns

• true if optimization was applied/verified, false if failed

reset_auto_optimization!()

Reset the automatic optimization state, forcing re-optimization on next data load.

show_auto_optimization_status()

Display the current status of automatic I/O optimization.

benchmark_buffer_sizes(simulation_path::String, output_num::Int; 
                      test_sizes=[32768, 65536, 131072, 262144], verbose=true)

Benchmark different buffer sizes to find the optimal setting for this specific simulation.

get_simulation_characteristics(simulation_path::String, output_num::Int)

Analyze simulation folder to determine optimal I/O settings. Returns a dictionary with simulation characteristics and recommended settings.

show_mera_cache_stats()

Print what the simulation-metadata cache currently holds: the number of entries and the path behind each one.

Use it to check whether a getinfo call was served from cache before reaching for clear_mera_cache!.

clear_mera_cache!()

Empty the cache of simulation metadata that Mera keeps between getinfo calls, and report how many entries were dropped.

Useful when a simulation folder changed on disk during a session and you want the next getinfo to re-read it rather than reuse what it saw earlier. See show_mera_cache_stats to inspect the cache first.