20150218-NOOR

Software bugs in Alcomo, LogMap and reference alignments: incoherence and semantic revision figures are incorrect; conclusion hold (see 20170208-NOOR)

Experiment design

This is a reproduction of 20140305-NOOR

Hypothesis: del < repl < add.

Experimental setting: As of 20140305-NOOR.

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2015-02-18

Lazy lavender hash: dd79573c717614724f74740d3d0004faab071c19

Parameters: params.sh

Command line (script.sh):

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-add.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=delete -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-del.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=replace -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-repl.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, 4*Ci7 2.3GHz, 16GB - MacOS X 9.5 - Java 1.7.0 HotSpot

Each simulation took around 4mn real time (it is done in 12mn)

Raw results

20150218-NOOR-Log4-2000-add.tsv 20150218-NOOR-Log4-2000-add.txt 20150218-NOOR-Log4-2000-del.tsv 20150218-NOOR-Log4-2000-del.txt 20150218-NOOR-Log4-2000-repl.tsv 20150218-NOOR-Log4-2000-repl.txt 20150218-NOOR-exec-log.txt

Result exploration

Success rate drawn in dotted; semantic F-measure in plain.
del (red), repl (green), add (blue)

modality	size	success	convergence	Incoherence degree	F-measure
Reference	70			0.00	1.00
Initial	54			0.47	0.20
Delete	11	0.93	960	0.0	0.21
Replace	11	0.89	1235	0.0	0.20
Add	14	0.89	1611	0.0	0.23
Alcomo	28		1	0.0	0.26
LogMap	39		1	0.0	0.25

Analysis

Key points:

The results are not quite those of 20140305-NOOR even while averaging on 10 run. It happens that:

del < repl < add is not strictly observed
the convergence rate and level is slower (for the three modalities)
the size of the resulting alignments is larger than for the previous experiment.

This is likely due to the generated situations which are more diverse than expected.

Further experiments: Redo this experiment with the same initial situations for the three modalities (this is not yet possible in the framework) to establish if del < repl < add is indeed true statistically and on every run.

This file can be retrieved from URL https://sake.re/20150218-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20150218-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)