20150218-NOOR

Software bugs in Alcomo, LogMap and reference alignments: incoherence and semantic revision figures are incorrect; conclusion hold (see 20170208-NOOR)

Experiment design

This is a reproduction of 20140305-NOOR

Hypothesis: del < repl < add.

Experimental setting: As of 20140305-NOOR.

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2015-02-18

Lazy lavender hash: dd79573c717614724f74740d3d0004faab071c19

Parameters: params.sh

Command line (script.sh):

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-add.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=delete -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-del.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=replace -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-repl.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, 4*Ci7 2.3GHz, 16GB - MacOS X 9.5 - Java 1.7.0 HotSpot

Each simulation took around 4mn real time (it is done in 12mn)

Raw results

20150218-NOOR-Log4-2000-add.tsv 20150218-NOOR-Log4-2000-add.txt 20150218-NOOR-Log4-2000-del.tsv 20150218-NOOR-Log4-2000-del.txt 20150218-NOOR-Log4-2000-repl.tsv 20150218-NOOR-Log4-2000-repl.txt 20150218-NOOR-exec-log.txt

Result exploration

modality size success convergence Incoherence degree F-measure
Reference 70 0.00 1.00
Initial 54 0.47 0.20
Delete 11 0.93 960 0.0 0.21
Replace 11 0.89 1235 0.0 0.20
Add 14 0.89 1611 0.0 0.23
Alcomo 28 1 0.0 0.26
LogMap 39 1 0.0 0.25

Analysis

Key points:

The results are not quite those of 20140305-NOOR even while averaging on 10 run. It happens that:

This is likely due to the generated situations which are more diverse than expected.

Further experiments: Redo this experiment with the same initial situations for the three modalities (this is not yet possible in the framework) to establish if del < repl < add is indeed true statistically and on every run.


This file can be retrieved from URL https://sake.re/20150218-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20150218-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr.