12. Scripting

We will focus on bash scripting here, for an example see the mkmh97.sh script. A typical script will contain the following components

  1. hash-bang the first line of the script, and optionally include nemo_functions.sh where various convenience functions are maintained

#! /usr/bin/env bash
#
source nemo_functions.sh

and make the script executable

chmod +x script_name

  1. parse the command line, ideally with a NEMO like keyword=value - optionally add qtrun directives to also allow to run it with a GUI frontend

run=run1       # directory and run ID       #> ENTRY
nbody=1000     # number of bodies           #> RADIO 1000,10000,100000,1000000
eps=0.05       # softening                  #> SCALE 0:1:0.01

and then parsing could be achieved as follows

for arg in "$@"; do
   export "$arg"
done

3. optionally, but highly recommended, some matching –HELP tags that can be retrieved using the –help command line option

#--HELP
a=1      # the a value      #> SCALE 0:10:1
b=2      # the b value      #> SCALE 0:20:2
#--HELP

if [ "$1" == "--help" ] || [ "$1" == "-h" ]; then
   set +x
   awk 'BEGIN{s=0} {if ($1=="#--HELP") s=1-s;  else if(s) print $0; }' $0
   exit 0
fi

  1. optionally maintain a set of nemopars or nemovar

_pars=nemopars.rc

save_vars="run nbody"

show_vars $save_vars >> $_pars
source  $_pars

echo "a1=$a1" >> $_pars

5. the body of the script, shell variables, nemopars variables etc. should be used where possible.

12.1. Example: Creating a runfile

A runfile is a simple text file, where each line in this file can be executed, for example using bash in a serial fashion. If each line is independant from all others, they can be executed in parallel using gnu parallel or sbatch (a common HPC batch queue system). For example the following command in NEMO produces a runfile with 4 lines:

mkrunfile.py progname a=1,2 b=2,4 c=10 > example1.run

progname a=1 b=2 c=10
progname a=2 b=2 c=10
progname a=1 b=4 c=10
progname a=2 b=4 c=10

where the example1.run file can be executed with any of the following commands. Depending on your resources of course. Memory, number of cores etc.

bash example1.run
parallel -j 4 < example1.run
sbatch_nemo.sh example1.run

in particular the last sbatch_nemo.sh example will likely need to be tailored for your sbatch based (HPC) system.

Note

Unless the parameters take care of this, you will need to ensure data are written to files that do not collide with each other. For example a directory or file that encodes the values of the parameters, or are numerically sorted (e.g. run010, run011) Currently mkrunfile.py does not have an automated way for this yet.

And here is an example of creating a runfile from a table with values

awk '{printf("progname runfile=run_%s a=%s b=%s\n", $1,$2,$3) }' example1.tab

12.2. Example: Extracting results from run directories

A common workflow is to run a series of simulations and walking over a multi-dimensional parameter space. This usually results into running each simulations in its own run-directory, plus storing parameters and possibly resulting values in a parameter file for later extraction. The directory name could even contain the value of these parameters as well. Several common strategies have been seen in the wild, we list three:

# 1. simply enumerated (e.g id=001,002,003,....) parameters stored inside
run_${id}

# 2. linear list of directories, parameters stored inside
run_${a}_${b}_${c}

# 3. hierarchical directories, parameters stored inside
run/$a/$b/$c

Although easier to visually identify the values of the parameters in 2. and 3., they don’t scale very well if a new parameter is introduced. In the first case a simple lookup table can be created using nemopars, thus making it easier to find which parameters are used in while run directory. Here’s an example:

nemopars id,a,b,c run_*/nemopars.rc  > run.pars

12.3. Summary

Summarizing, here are the recommended methods to maintain and extract NEMO variables.

  1. nemopars: extract parameters from a bash-style rc file (python should also be able to use it)

  2. nemovar: get and set NEMOVAR variables

  3. show_vars: alias via nemo_functions.sh