20140311-NOOR

Experiment design

Hypothesis: Confirm the hypothesis that in 20140306-NOOR the difference between LogMap and Alcomo and the AlignmentAdjustingAgent approach cannot be found in simple terms of precision and recall, but are due to the semantic precision and recall.

Experimental setting: Very same setting as 20140306-NOOR with classical precision and recall, but the lines implementing semantic precision and recall have been replaced by those with classical precision and recall (commented in the software)

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2014-03-11

Lazy lavender hash: 2aec5fe496c2b95760dba0ef87e82ac13264879b

Parameters: params.sh

Command line (script.sh):

 java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=delete -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-del.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=replace -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-repl.txt

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=4 -DnbIterations=2000 > log4-2000-add.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

20140311-NOOR-4add.tsv 20140311-NOOR-4add.txt 20140311-NOOR-4del.tsv 20140311-NOOR-4del.txt 20140311-NOOR-4repl.tsv 20140311-NOOR-4repl.txt

Result exploration

Top:

Success rate drawn in dotted; syntactic F-measure in plain.
del (red), repl (green), add (blue)

Bottom:

dotted: add success rate;
dashed: #correspondences for add
plain: syntactic F-measure for add
black: syntactic F-measure for LogMap/Alcomo

modality	size	success	convergence	Incoherence degree	F-measure
Reference	70			0.00	1.00
Initial	54			0.49	0.20
Delete	6	0.98	330	0.0	0.16
Replace	6	0.95	860	0.0	0.16
Add	12.2	0.88	1280	0.0	0.16
Alcomo	27.3		1	0.0	0.14
LogMap	35.7		1	0.0	0.14

Analysis

Key points: The hypothesis is confirmed:

Measures are far lower for all repair mechanisms;
With syntactic precision and recall, the add modalities (and in fact all the others) are better than the baseline algorithms.

Further experiments:

This file can be retrieved from URL https://sake.re/20140311-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140311-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)