20140305-NOOR

Software bugs in Alcomo, LogMap and reference alignments: incoherence and semantic revision figures are incorrect; conclusion hold (see 20170208-NOOR)

Experiment design

Hypothesis: del < repl < add.

Experimental setting: A remake of 20140225-NOOR, except that:

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2014-03-05

Lazy lavender hash: 2aec5fe496c2b95760dba0ef87e82ac13264879b

Parameters: params.sh

Command line (script.sh):

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-add.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=delete -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-del.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=replace -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-repl.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

20140305-NOOR-Log4-2000-add.tsv 20140305-NOOR-Log4-2000-add.txt 20140305-NOOR-Log4-2000-del.tsv 20140305-NOOR-Log4-2000-del.txt 20140305-NOOR-Log4-2000-repl.tsv 20140305-NOOR-Log4-2000-repl.txt

Result exploration

modality size success convergence Incoherence degree F-measure
Reference 70 0.00 1.00
Initial 54 0.49 0.20
Delete 6 0.98 400 0.0 0.16
Replace 6 0.95 1000 0.0 0.16
Add 12.7 0.89 1330 0.0 0.23
Alcomo 25.5 1 0.0 0.26
LogMap 36.5 1 0.0 0.26

Analysis

Key points:

As expected, del < repl < add.

Further experiments:


This file can be retrieved from URL https://sake.re/20140305-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140305-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr.