Sake.re: metadata

Besides the content of the script.sh most of the experiments are described through metadata. These metadata, in turn, are used for various purposes:

• Describing the experiment to human beings;
• Guiding the processing and its output;
• Being displayed by the analysis;
• Generating metadata for publication sites;
• Tagging the status of experiments.

Metadata types

These metadata are entered in the params.sh file which only contains shell variable assignments. This may look as a strange way to express metadata: it would certainly be better to use semantic (RDF) descriptions of experiments. However, nothing seems to be ready for that, so let stick to this one for the moment. Beware that these variables cannot be assigned any values like in a shell, because they may be included in the Docker file, which only evaluates variables at build time or through Jupyter notebooks which are not shells.

Below is a short and non exhaustive list of metadata (C=compulsory, O=optional): The params.sh file may contain more variables which can used to document better the experiments. These metadata can be classified into three stantard categories (plus one non standard one). They could also be classified with respect to a further consideration of Design, Execution, Analysis.

Environment metadata

Environment metadata (E) describes the environment required by the experiment (operating system and software versions, etc.). It is very useful for repeating and reproducing experiments.

• OSVERS (CEE): the version of the operating system to run the experiment on (Docker);
• LOGBACK_VERSION (OEE): the version of the logback logger implementation;
• JPATH (CEE): the classpath used when running the experiment;
• LLHASH (CEE): the hash of the version of Lazy lavender to be used;
• LABEL (CEE): a unique label following the pattern YYYYMMDD-TYPE identifying the experiment;

Experiment metadata

Experiment metadata (X) describes the parameters of the experiment (type of experiment, number of agents, type of collected output, etc.). They describe the design of an experiment and in particular its experimental plan.

• NAME (CXD): a name for the experiment (may not be used);
• OPTS (OXD): the options share by all runs to be passed to Lazy lavender;
• LOADOPT (OXE): the loading option (for reloading a sequence of event);
• DIRPREF (OXE): the subdirectories where to store experiment results;
• TOCOPY (CXE): the files to copy from the current directory in a Docker container;
• OUTPUT (CXE): the base directory where results are stored (advised results/);
• CONTROLED (CXD): the varying controlled variables;
• FIXED (CXD): the fixed controlled variables;
• OBSERVED (CXD): the directly observed variables;
• DERIVED (ODA): variables derived from observed variables.

Experiment-specific metadata

This is completed by non-standard variables commonly used:

• NBRUNS (OXE): number of runs (values of nbRuns);
• NBITERATIONS (OXE): number of iterations (values of nbIterations);
• NBAGENTS (OXE): number of agents (values of nbAgents);
• NBPOPS (OXE): number of populations (values of nbPops);
• OPS (OXE): operators (values of revisionModalities);
• NBCORRESP (OXE): initial size of the alignments (values of nbCorresp);
• DISJOINTRATIO (OXE): relaxation percentage (values of disjointRatio);

Documentation metadata (D)

Documentation metadata (D) provides information used for documenting the experiment (dates, performers, hypothesis tested, summary, etc.). In principle, they are not necessary to reproduce the experiment. They are however used by the notebook template to quickly generate a notebook.

• EXPE (CDD): description of the experiment;
• TITLE (CDD): more informative description of the experiment;
• ABSTRACT (ODD): an even more informative and complete description of the experiment used as publication presentation;
• HYPOTHESIS (CDD): the hypothesis/hypotheses that this experiment is supposed to test;
• HARDWARE (ODE): the hardware on which the experiment has been performed;
• OS (ODE): the operating system on which the experiment has been performed;
• RESULT (CDA):
• STATUS (CDA): Describe the state of the experiments which may contain the following value:
  • DESIGNED: The experiment has been designed but not performed;
  • PERFORMED: The experiment has been performed but not analysed;
  • VALID: The experiment has been analysed and its results hold;
  • SUBSUMED: The experiment has been repeated;
  • PARTLY: The experiment results are only partialy holding;
  • UNCERTAIN: The experiment results are under scrutiny and may be invalidated;
  • INVALID: The experiment results are invalid.
• STATUSDESC (ODA): This depends on the value of STATUS. If it is SUBSUMED then it contains the LABEL of the subsuming experiment withing brackets; if it is INVALID, PARTLY or UNCERTAIN then it contains a sentence explaining the status.
• SETTING (CDD): a description of the experiment process;
• VARIATIONOF (ODD): the name of an experiment this one is based on;
• CLASSES (CDD): the classes used in this experiment;
• DESIGNDATE (ODD): the date the experiment was designed;
• DESIGNER (CDD): the name of the designer;
• DATE (CDE): the date the experiment was performed;
• EXPERIMENTER (CDE): the name of the experimenter;
• ANALYST (CDA): the name of the analyst;

Example

Here is the file provided with the experiment template (YYYYMMDD-TYPE), which is fully functional:

# ======================================== ENVIRONMENT

OSVERS=busternb

LOGBACK_VERSION=1.2.3

JPATH=lazylav/lib/lazylav/ll.jar:lib/slf4j/logback-classic-${LOGBACK_VERSION}.jar:lib/slf4j/logback-core-${LOGBACK_VERSION}.jar:.

LLHASH=b2093e6ce8e1d2aeddc5e08d79eb6a08d1932894

# ======================================== EXPERIMENT

LABEL=YYYYMMDD-TYPE
NAME=${LABEL}
DESIGNDATE=2020-12-23

# Experiment specific
CONTROLED="revisionModality"
FIXED="nbAgents nbRuns nbIterations"
OBSERVED="srate size prec rec"

NBAGENTS=4
NBITERATIONS=100
NBRUNS=3
NBPOPS=4
SYNCRATE=250
OPS="delete replace refine add addjoin refadd"

OPT="-DnbAgents=${NBAGENTS} -DnbIterations=${NBITERATIONS} -DnbRuns=${NBRUNS} -DreportPrecRec" 

LOADOPT=
DIRPREF=${NBAGENTS}-${NBITERATIONS}

TOCOPY="params.sh script.sh notebook.ipynb logback.xml"

# !! should better be results/
OUTPUT=results/

# ======================================== METADATA

VARIATIONOF=20191218-NOOR
EXPE="Put a short description of the experiment"

HYPOTHESIS="A precise and short description of tested hypotheses"

DATE=2020-12-23

SETTING="Variation of ${VARIATIONOF} with a different setting (describe)"

CLASSES="Population, PopulatedARExperiment, PopulationAlignmentAdjustingAgent, NOOEnvironment, ActionLogger, AverageLogger, Monitor"

# ======================================== STATUS

RESULT="One line statement describing the obtained results of this experiment"
# One of: DESIGNED PERFORMED VALID SUBSUMED PARTLY UNCERTAIN INVALID
STATUS=DESIGNED
#STATUSDESC

PERFORMER="Your name here (affiliation)"
DESIGNER=${PERFORMER}
EXPERIMENTER=${PERFORMER}
ANALYST=${PERFORMER}