20180311-NOOR

Experiment design

Hypothesis:

• reproduction of results of 20170215a-NOOR
• clearer behaviour of add/addjoin after bug correction (3fcf33/2018-03-05) affecting addjoin/refadd

Experimental setting: Same as the initial experiments 20170215a-NOOR with:

• fixed errors in the implementation of addjoin (affects addjoin and refadd)
• comparison on the exact same set of games (replayGames)

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2018-03-11

Lazy lavender hash: b86d0cb89108ca78d0ead507098e14683e4d39fb

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

Parameters: params.sh

Command line (script.sh):

JPATH=lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.
OPT="-DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec"

# Dry test only for generating the runs
# bash scripts/runexp.sh -d 4-10000-nothing java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=nothing -DsaveDir=input/expeRun -DsaveInit -DsaveParams -DsaveGames

LOADOPT="-DloadDir=input/expeRun -DloadEnv -DloadAgents -DreplayGames"

for mod in delete replace refine add addjoin refadd
do
   bash scripts/runexp.sh -d 4-10000-${mod} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${mod} -DimmediateRatio=80
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_151-b12 HotSpot

Note: This experiment has been set up to first generate the runs and then play them with different configuration. To strictly repeat it, it is necessary to reuse the same input and not generate a new one. Hence, the input is saved in the repository, the generating line above must be commented (including in Docker).

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun.zip Then unzip expeRun.zip -d input

Raw results

4-10000-add-im80.tsv 4-10000-add-im80.txt 4-10000-addjoin-im80.tsv 4-10000-addjoin-im80.txt 4-10000-delete-im80.tsv 4-10000-delete-im80.txt 4-10000-refadd-im80.tsv 4-10000-refadd-im80.txt 4-10000-refine-im80.tsv 4-10000-refine-im80.txt 4-10000-replace-im80.tsv 4-10000-replace-im80.txt

Result exploration

• In plain: success rate;
• In dashed: F-measure.

• In plain: precision;
• In dashed: recall.

Observations:

• Note that the set of game events are the same as those of 20180308-NOOR

Analysis

Key points:

• The results are commensurate with 20170215a-NOOR
• Except that now, as expected, final results for add and addjoin are the same
• As a consequence results of refadd are even better (they use addjoin)
• Applying the same set of events shows that addjoin converges faster, but add reaches the same resulting alignments
• The results have the same relation with 20180308-NOOR (smaller size, 100% precision, lower recall) as 20170215a-NOOR has with 20170214a-NOOR and 20170214b-NOOR

Further experiments:

• Redo all experiments with expansion, generative and start empty.

Remarks:

• Now, again the convergence of add and addjoin are different (addjoin converging faster as expected)

This file can be retrieved from URL https://sake.re/20180311-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180311-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180311-LOG.zip