20171128-NOOR

2018-09-17: expansion did not work properly since commit 562f5c4b96893d30e2dc9fd59d935e013ce5219f (see 20180917-NOOR)

Experiment design

Hypothesis:

• The trends observed in 20170530-NOOR and 20170531-NOOR do not depend on the number of agents

Experimental setting: After 20171001-NOOR, we estimated that the convergence should be reached by 45000. Hence the experimental setting is the same as 20171001-NOOR with 45000 iterations

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2017-11-28 to 2017-12-10

Lazy lavender hash: 9af56486cfe565bca783c866ed9c0f7d4ca520e5

Parameters: params.sh

Command line (script.sh):

for op in delete replace refine add addjoin refadd
do
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=45000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=45000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=45000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=45000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=45000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative -Dstartempty
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=45000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative -Dstartempty
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_131-b11 HotSpot

Raw results

5-45000-add-clever-nr-gen-empty.tsv 5-45000-add-clever-nr-gen-empty.txt 5-45000-add-clever-nr-gen.tsv 5-45000-add-clever-nr-gen.txt 5-45000-add-clever-nr-im80-gen-empty.tsv 5-45000-add-clever-nr-im80-gen-empty.txt 5-45000-add-clever-nr-im80-gen.tsv 5-45000-add-clever-nr-im80-gen.txt 5-45000-add-clever-nr-im80.tsv 5-45000-add-clever-nr-im80.txt 5-45000-add-clever-nr.tsv 5-45000-add-clever-nr.txt 5-45000-addjoin-clever-nr-gen-empty.tsv 5-45000-addjoin-clever-nr-gen-empty.txt 5-45000-addjoin-clever-nr-gen.tsv 5-45000-addjoin-clever-nr-gen.txt 5-45000-addjoin-clever-nr-im80-gen-empty.tsv 5-45000-addjoin-clever-nr-im80-gen-empty.txt 5-45000-addjoin-clever-nr-im80-gen.tsv 5-45000-addjoin-clever-nr-im80-gen.txt 5-45000-addjoin-clever-nr-im80.tsv 5-45000-addjoin-clever-nr-im80.txt 5-45000-addjoin-clever-nr.tsv 5-45000-addjoin-clever-nr.txt 5-45000-delete-clever-nr-gen-empty.tsv 5-45000-delete-clever-nr-gen-empty.txt 5-45000-delete-clever-nr-gen.tsv 5-45000-delete-clever-nr-gen.txt 5-45000-delete-clever-nr-im80-gen-empty.tsv 5-45000-delete-clever-nr-im80-gen-empty.txt 5-45000-delete-clever-nr-im80-gen.tsv 5-45000-delete-clever-nr-im80-gen.txt 5-45000-delete-clever-nr-im80.tsv 5-45000-delete-clever-nr-im80.txt 5-45000-delete-clever-nr.tsv 5-45000-delete-clever-nr.txt 5-45000-refadd-clever-nr-gen-empty.tsv 5-45000-refadd-clever-nr-gen-empty.txt 5-45000-refadd-clever-nr-gen.tsv 5-45000-refadd-clever-nr-gen.txt 5-45000-refadd-clever-nr-im80-gen-empty.tsv 5-45000-refadd-clever-nr-im80-gen-empty.txt 5-45000-refadd-clever-nr-im80-gen.tsv 5-45000-refadd-clever-nr-im80-gen.txt 5-45000-refadd-clever-nr-im80.tsv 5-45000-refadd-clever-nr-im80.txt 5-45000-refadd-clever-nr.tsv 5-45000-refadd-clever-nr.txt 5-45000-refine-clever-nr-gen-empty.tsv 5-45000-refine-clever-nr-gen-empty.txt 5-45000-refine-clever-nr-gen.tsv 5-45000-refine-clever-nr-gen.txt 5-45000-refine-clever-nr-im80-gen-empty.tsv 5-45000-refine-clever-nr-im80-gen-empty.txt 5-45000-refine-clever-nr-im80-gen.tsv 5-45000-refine-clever-nr-im80-gen.txt 5-45000-refine-clever-nr-im80.tsv 5-45000-refine-clever-nr-im80.txt 5-45000-refine-clever-nr.tsv 5-45000-refine-clever-nr.txt 5-45000-replace-clever-nr-gen-empty.tsv 5-45000-replace-clever-nr-gen-empty.txt 5-45000-replace-clever-nr-gen.tsv 5-45000-replace-clever-nr-gen.txt 5-45000-replace-clever-nr-im80-gen-empty.tsv 5-45000-replace-clever-nr-im80-gen-empty.txt 5-45000-replace-clever-nr-im80-gen.tsv 5-45000-replace-clever-nr-im80-gen.txt 5-45000-replace-clever-nr-im80.tsv 5-45000-replace-clever-nr-im80.txt 5-45000-replace-clever-nr.tsv 5-45000-replace-clever-nr.txt

Result exploration

• In green: 100% precision, i.e., a network with no error
• In orange: last failure (at least in one of the 10 runs) over 40000 games
• In red: last failure (at least in one of the 10 runs) over 44000 games

(drawn from data scaled every 100 measures)

Analysis

Key points:

• The behaviour or 'add' is highly different, hence must be investigated. Actually, the way operators have been designed predicts that 'add' is slower to converge than 'addjoin'. However, looking at the curves it is a clear outlier.
• Some other situations: clever-nr-gen do not seem to converge towards 100% precision at that stage;
• The trends of 20170530-NOOR and 20170531-NOOR are observed, except that starting empty leads to more quickly to correct correspondence when im80 is not used
• Recall and thus F-measure are lower than with 4 agents (this is inline with the observations of 20150219-NOOR)

Further experiments:

• Redo some of these experiments with a larger number of games in order to determine if this is simply a slow convergence or a lack of convergence (see 20171218-NOOR).
• Check the acurracy of the 'add' behaviour

This file can be retrieved from URL https://sake.re/20171128-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20171128-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20171128-NOOR.zip