20140225-NOOR

Experiment design

Hypothesis:

The add modality reaches a better F-measure than replace which reaches a better F-measure than delete.
The add modality on 4 agents outperforms LogMap and Alcomo

Experimental setting: This is the same setting as 20140224-NOOR, except that:

the exercise run 10 times for 500 iteration

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2014-02-25

Lazy lavender hash: 402eae910f33cf73a4663fe83308d0b0be6d7b5b

Parameters: params.sh

Command line (script.sh):

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=delete -DnbRuns=10 -DnbAgents=4 -DnbIterations=500 -o ~/plot12del.table 

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=replace -DnbRuns=10 -DnbAgents=4 -DnbIterations=500 -o ~/plot12repl.table 

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=4 -DnbIterations=500 -o ~/plot12add.table

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

20140225-NOOR.txt

Result exploration

Plain lines are success rates; dashed lines are F-measure
In blue, add;
In green, replace;
In red, delete;
In black, the Alcomo baseline.

modality	success	convergence	Incoherence degree	F-measure
Initial			0.49	0.20
Delete	0.90	445	0.0	0.16
Replace	0.81	489	0.0	0.18
Add	0.62	500+	0.0	0.26
LogMap		1	0.0	0.25
Alcomo		1	0.0	0.26

Analysis

Key points:

The situation is better than with three agents in which case very often the revision deletes all the correspondences which are not top=top
Clearly add outperforms replace which is slightly better than delete
Add has not converged after 500 iterations (on 10 runs), it takes around 1200 iterations
Again, the baseline is reached, but not improved.

Among the runs, there is a lot of variety, any of the system may reach the highest F-measure (with the add modality).

Further experiments:

Further increase the number of agents
Try with more elaborate modalities

This file can be retrieved from URL https://sake.re/20140225-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140225-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)