20140305-NOOR

Software bugs in Alcomo, LogMap and reference alignments: incoherence and semantic revision figures are incorrect; conclusion hold (see 20170208-NOOR)

Experiment design

Hypothesis: del < repl < add.

Experimental setting: A remake of 20140225-NOOR, except that:

The number of iterations is 2000.

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2014-03-05

Lazy lavender hash: 2aec5fe496c2b95760dba0ef87e82ac13264879b

Parameters: params.sh

Command line (script.sh):

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-add.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=delete -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-del.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=replace -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-repl.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

20140305-NOOR-Log4-2000-add.tsv 20140305-NOOR-Log4-2000-add.txt 20140305-NOOR-Log4-2000-del.tsv 20140305-NOOR-Log4-2000-del.txt 20140305-NOOR-Log4-2000-repl.tsv 20140305-NOOR-Log4-2000-repl.txt

Result exploration

Success rate drawn in dotted; semantic F-measure in plain.
del (red), repl (green), add (blue)

modality	size	success	convergence	Incoherence degree	F-measure
Reference	70			0.00	1.00
Initial	54			0.49	0.20
Delete	6	0.98	400	0.0	0.16
Replace	6	0.95	1000	0.0	0.16
Add	12.7	0.89	1330	0.0	0.23
Alcomo	25.5		1	0.0	0.26
LogMap	36.5		1	0.0	0.26

Analysis

Key points:

As expected, del < repl < add.

Further experiments:

This file can be retrieved from URL https://sake.re/20140305-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140305-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)