Sake.re: metadata

Besides the content of the script.sh most of the experiments are described through metadata. These metadata, in turn, are used for various purposes:

They will facilitate all tasks. Hence, it is worth devoting some time to fill them.

Metadata types

These metadata are entered in the params.sh file which only contains shell variable assignments. This may look as a strange way to express metadata: it would certainly be better to use semantic (RDF) descriptions of experiments. However, nothing seems to be ready for that, so let stick to this one for the moment. Beware that these variables cannot be assigned any values like in a shell, because they may be included in the Docker file, which only evaluates variables at build time or through Jupyter notebooks which are not shells.

Below is a short and non exhaustive list of metadata (C=compulsory, O=optional): The params.sh file may contain more variables which can used to document better the experiments. These metadata can be classified into three stantard categories (plus one non standard one). They could also be classified with respect to a further consideration of Design, Execution, Analysis.

Environment metadata

Environment metadata (E) describes the environment required by the experiment (operating system and software versions, etc.). It is very useful for repeating and reproducing experiments.

Experiment metadata

Experiment metadata (X) describes the parameters of the experiment (type of experiment, number of agents, type of collected output, etc.). They describe the design of an experiment and in particular its experimental plan.

Experiment-specific metadata

This is completed by non-standard variables commonly used:

Documentation metadata (D)

Documentation metadata (D) provides information used for documenting the experiment (dates, performers, hypothesis tested, summary, etc.). In principle, they are not necessary to reproduce the experiment. They are however used by the notebook template to quickly generate a notebook.

Example

Here is the file provided with the experiment template (YYYYMMDD-TYPE), which is fully functional:

# ======================================== ENVIRONMENT

OSVERS=busternb

LOGBACK_VERSION=1.2.3

JPATH=lazylav/lib/lazylav/ll.jar:lib/slf4j/logback-classic-${LOGBACK_VERSION}.jar:lib/slf4j/logback-core-${LOGBACK_VERSION}.jar:.

LLHASH=b2093e6ce8e1d2aeddc5e08d79eb6a08d1932894

# ======================================== EXPERIMENT

LABEL=YYYYMMDD-TYPE
NAME=${LABEL}
DESIGNDATE=2020-12-23

# Experiment specific
CONTROLED="revisionModality"
FIXED="nbAgents nbRuns nbIterations"
OBSERVED="srate size prec rec"

NBAGENTS=4
NBITERATIONS=100
NBRUNS=3
NBPOPS=4
SYNCRATE=250
OPS="delete replace refine add addjoin refadd"

OPT="-DnbAgents=${NBAGENTS} -DnbIterations=${NBITERATIONS} -DnbRuns=${NBRUNS} -DreportPrecRec" 

LOADOPT=
DIRPREF=${NBAGENTS}-${NBITERATIONS}

TOCOPY="params.sh script.sh notebook.ipynb logback.xml"

# !! should better be results/
OUTPUT=results/

# ======================================== METADATA

VARIATIONOF=20191218-NOOR
EXPE="Put a short description of the experiment"

HYPOTHESIS="A precise and short description of tested hypotheses"

DATE=2020-12-23

SETTING="Variation of ${VARIATIONOF} with a different setting (describe)"

CLASSES="Population, PopulatedARExperiment, PopulationAlignmentAdjustingAgent, NOOEnvironment, ActionLogger, AverageLogger, Monitor"

# ======================================== STATUS

RESULT="One line statement describing the obtained results of this experiment"
# One of: DESIGNED PERFORMED VALID SUBSUMED PARTLY UNCERTAIN INVALID
STATUS=DESIGNED
#STATUSDESC

PERFORMER="Your name here (affiliation)"
DESIGNER=${PERFORMER}
EXPERIMENTER=${PERFORMER}
ANALYST=${PERFORMER}