Sake.re: Producing (new) experiments

See the prerequisite and be sure to be familiar with Sake.re.

Designing experiments

This starts by first cloning the experiment template:

$ git clone --recurse-submodules cakes@felapton.inrialpes.fr:YYYYMMDD-TYPE.git ${LABEL}
$ cd ${LABEL}
$ git remote remove origin
It is advised to upgrade supporting software (Lazy lavender and pySake) by:
$ git submodule foreach 'git checkout master'
$ git commit -a -m "Upgraded submodule versions"

Then, edit the experiment descriptions params.sh, script.sh, notebook.ipynb and eventually the input directory.

Here you are on your own, this is where you can carefully design your own experiments. This is of course the main part of the work and it is fully dependent on your own needs.

Once this done, tag the repository with the DESIGNED label.

$ git commit -a -m "Explain the design"
$ git tag DESIGNED -m "Experiment design completed"

Running experiments

Compile lazy lavender:

$ ant -f lazylav/build.xml compileall
Process the experiment:
$ bash script.sh

It may be useful for later to generate the docker file (this actually can be done already at design time):

$ lazylav/scripts/gendkr.sh params.sh script.sh
$ git add ${LABEL}.dkr
$ git commit -a -m "Dockerfile generated"
The variables that have to be set in params.sh before running this are: LABEL, OSVERS, OUTPUT=results/, LLHASH, and LOGBACK_VERSION.

Then tag the repository with the PERFORMED label.

$ git tag PERFORMED -m "Experiment performed"

Analysing experiments

You must have python and jupyter notebook installed. If not already installed, this can be achieved with:

$ python3 -m pip install notebook
$ python3 -m pip install jupyter_contrib_nbextensions
$ jupyter nbextension enable python-markdown/main
$ jupyter nbextension enable hide_input_all/main
$ jupyter nbextension enable collapsible_headings/main
$ jupyter nbextension enable livemdpreview/livemdpreview

You can refine the notebook.ipynb notebook through jupyter, by running:

$ jupyter trust notebook.ipynb
$ jupyter notebook
And changing it through the browser usually at the http://localhost:8888 address.

Before committing the notebook file, strip it from its output so that git only tracks the input cells:

$ python3 pysake/stripvar.py notebook.ipynb > notebook.tmp.ipynb
$ jupyter nbconvert --ClearOutputPreprocessor.enabled=True notebook.tmp.ipynb --to=notebook --output=notebook.ipynb

Once the analysis completed, it is necessary to generate an HTML file from it that can be published without jupyter:

$ jupyter nbconvert --execute -y notebook.ipynb --to html --ExecutePreprocessor.timeout=-1 --TemplateExporter.exclude_input=True --output index.html

It may also be useful to record the set of python packages necessary to the experiment in a requirements.txt file. This can be done with virtual environments.

Finally commit and tag:

$ git add analysis requirements.txt index.html                      # not 'analysis' anymore?
$ git commit -a -m "analyzed"
$ git tag ANALYSED -m "Analysis completed"
Methodologically, preparing the analysis at design time, is even better.


You are encouraged to publish your experiment results. If you have followed the instructuon above, you should be well prepared for that.