20140207-NOOR

Invalid experiment: incorrect reference alignment (see 20140208-NOOR)

Experiment design

Hypothesis: The network can achieve null incoherence degree.

The network can revise better (in terms of F-measure) than LogMap and Alcomo.

Experimental setting: Same as 20140204-NOOR but it is possible to compare with LogMap and Alcomo.

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2014-02-07

Lazy lavender hash: 279920d1b24c5c799c7ba550402d1a6aabf2a11d

Parameters: params.sh

Command line (script.sh):

java -Xms500M -Xmx1G -Dlog.level=DEBUG -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DreportIncoherence=1

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

20140207-NOOR.tsv 20140207-NOOR.txt

Result exploration

In blue the average success rate (over 5 runs);
In dotted, the minimum and maximum success rate (over the same 5 runs);
In red, the average syntactic F-measure (Alignment API).
In green, the average incoherence degree (Alcomo).
In black, the logMap and Alcomo baseline F-measure.

Average degrees:

measure	F-measure	Incoherence degree
Initial	0.12	0.33
LogMap	0.13	0.18
Alcomo	0.13	0.00
Final	0.15	0.01

Analysis

After, 100+ steps the network has reduced incoherence to nearly null.

Key points:

F-measure passes above that of LogMap and Alcomo (simply because agents know exactly what to revise).

F-measure remains low because:

All actions only remove correspondences, but above all
F-measure is purely syntactic
Alignments are not symmetric, then revision can only apply on half of them

Incoherence degree is non null because:

Alignments are not symmetric

Further experiments:

Make the matrix symmetric or implements "alignment" at each stage
Use semantic precision and recall.

This file can be retrieved from URL https://sake.re/20140207-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140207-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)