20180308-NOOR

Experiment design

Hypothesis:

• reproduction of results of 20170214a-NOOR and 20170214b-NOOR
• clearer behaviour of add/addjoin after bug correction (3fcf33/2018-03-05) affecting addjoin/refadd
• clarification of relations between add and addjoin (by replaying the same set of events)

Experimental setting: Same as the initial experiments 20170214a-NOOR and 20170214b-NOOR with:

• after fixed errors in the implementation of addjoin (affects addjoin and refadd)
• with comparison on the exact same set of games (replayGames)

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2018-03-08

Lazy lavender hash: fa69a4569fb2ecd7d8f24759d1ce0cd71080811a

Command line (script.sh):

JPATH=lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.
OPT="-DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec"

# Dry test only for generating the runs
# bash scripts/runexp.sh -d 4-10000-nothing java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=nothing -DsaveDir=input/expeRun -DsaveInit -DsaveParams -DsaveGames

LOADOPT="-DloadDir=input/expeRun -DloadEnv -DloadAgents -DreplayGames"

for mod in delete replace refine add addjoin refadd
do
   bash scripts/runexp.sh -d 4-10000-${mod} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${mod}
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_151-b12 HotSpot

Note: This experiment has been set up to first generate the runs and then play them with different configuration (it has generated the expeRun dataset). To strictly repeat it, it is necessary to reuse the same input and not generate a new one. Hence, the input is saved in the repository, the generating line above must be commented (including in Docker).

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun.zip Then unzip expeRun.zip -d input

Raw results

4-10000-add.tsv 4-10000-add.txt 4-10000-addjoin.tsv 4-10000-addjoin.txt 4-10000-delete.tsv 4-10000-delete.txt 4-10000-nothing.tsv 4-10000-nothing.txt 4-10000-refadd.tsv 4-10000-refadd.txt 4-10000-refine.tsv 4-10000-refine.txt 4-10000-replace.tsv 4-10000-replace.txt

Result exploration

• In plain: success rate;
• In dashed: F-measure.

• In plain: precision;
• In dashed: recall.

• In red: add; in blue: addjoin;
• In plain: success rate;
• In dashed: F-measure.

Analysis

Key points:

• The results are commensurate with 20170214a-NOOR and 20170214b-NOOR
• Except that now, as expected, final results for add and addjoin are the same
• As a consequence results of refadd are even better (they used addjoin)
• Applying the same set of events shows that addjoin converges faster, but add reaches the same resulting alignments

Further experiments:

• Redo all experiments with expansion, relaxation, generative and start empty.

Remarks:

• The fact that the worse case convergence for addjoin is the same as for add may simply be related to the case of the last converging example out of 10. To be checked.

This file can be retrieved from URL https://sake.re/20180308-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180308-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180308-LOG.zip