20140304-NOOR

Experiment design

Hypothesis: If one leaves the process to last long enough, it will provide a better F-measure than logical repair.

Experimental setting: As 20140225-NOOR, except that:

The number of iterations is 10000;
The number of agents goes from 3 to 6;
The software now logs the total number of correspondences.

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2014-03-04

Lazy lavender hash: 2aec5fe496c2b95760dba0ef87e82ac13264879b

Parameters: params.sh

Command line (script.sh):

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=6 -DnbIterations=10000 > log6-10000.txt 

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=3 -DnbIterations=10000 > log3-10000.txt 

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=5 -DnbIterations=10000 > log5-10000.txt 

java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=4 -DnbIterations=10000 > log4-10000.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Takes a whole night.

Raw results

20130304-NOOR-Log3-10000.tsv 20130304-NOOR-Log3-10000.txt 20130304-NOOR-Log4-10000.tsv 20130304-NOOR-Log4-10000.txt 20130304-NOOR-Log5-10000.tsv 20130304-NOOR-Log5-10000.txt 20130304-NOOR-Log6-10000.tsv 20130304-NOOR-Log6-10000.txt

Result exploration

In dotted, success rate;
In dashed, number of correspondences;
In plain, semantic F-measure;
In black, baseline semantic F-measure.

nbagents	#ref	#init	#logmap	#alcomo	#final	!init	F-init	F-logmap	F-alcomo	F-final	conv
3	15	15	12	10.3	3	.31	.32	.35	.36	.33	300
4	70	54	36.7	28.4	12.4	.47	.20	.24	.25	.21	1670
5	250	170	94.7	71.7	47.4	.58	.11	.18	.17	.24	5400
6	783	495	234	182	224	.63	.06	.12	.11	.14	10000+

* #= number of correspondences * != incoherence rate * F= semantic F-measure * conv= game at which convergence is reached

Analysis

Key points:

The behaviour is long and erratic
In the long run, it seems that indeed average F-measure is better.

Further experiments:

Complement: On 2014-03-11, a supplementary 10 runs with 5 agents have been performed for confirmation purposes. It provided the very same results.

This file can be retrieved from URL https://sake.re/20140304-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140304-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)