20171218-NOOR

2018-05-16: Under scrutiny because expansion did not work since commit 562f5c4b96893d30e2dc9fd59d935e013ce5219f

Experiment design

Hypothesis:

• With 5 agents clever-nr-gen and add indeed converges (see 20171128-NOOR)

Experimental setting: Same as 20171128-NOOR with 70000 and 100000 iterations only on:

• add
• clever-nr-gen (expansion + generation)

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2017-12-18 to 2017-12-30

Lazy lavender hash: ab0edf530d7d832b2da9b6f208f729917fbe8e59

Parameters: params.sh

Command line (script.sh):

for it in 70000 100000
do 
   for op in delete replace refine add addjoin refadd
   do
      java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative
   done
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -Dgenerative -Dstartempty
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative -Dstartempty
  done
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_131-b11 HotSpot

Raw results

5-100000-add-clever-nr-gen-empty.tsv 5-100000-add-clever-nr-gen-empty.txt 5-100000-add-clever-nr-gen.tsv 5-100000-add-clever-nr-gen.txt 5-100000-add-clever-nr-im80-gen-empty.tsv 5-100000-add-clever-nr-im80-gen-empty.txt 5-100000-add-clever-nr-im80-gen.tsv 5-100000-add-clever-nr-im80-gen.txt 5-100000-addjoin-clever-nr-gen.tsv 5-100000-addjoin-clever-nr-gen.txt 5-100000-delete-clever-nr-gen.tsv 5-100000-delete-clever-nr-gen.txt 5-100000-refadd-clever-nr-gen.tsv 5-100000-refadd-clever-nr-gen.txt 5-100000-refine-clever-nr-gen.tsv 5-100000-refine-clever-nr-gen.txt 5-100000-replace-clever-nr-gen.tsv 5-100000-replace-clever-nr-gen.txt 5-70000-add-clever-nr-gen-empty.tsv 5-70000-add-clever-nr-gen-empty.txt 5-70000-add-clever-nr-gen.tsv 5-70000-add-clever-nr-gen.txt 5-70000-add-clever-nr-im80-gen-empty.tsv 5-70000-add-clever-nr-im80-gen-empty.txt 5-70000-add-clever-nr-im80-gen.tsv 5-70000-add-clever-nr-im80-gen.txt 5-70000-addjoin-clever-nr-gen.tsv 5-70000-addjoin-clever-nr-gen.txt 5-70000-delete-clever-nr-gen.tsv 5-70000-delete-clever-nr-gen.txt 5-70000-refadd-clever-nr-gen.tsv 5-70000-refadd-clever-nr-gen.txt 5-70000-refine-clever-nr-gen.tsv 5-70000-refine-clever-nr-gen.txt 5-70000-replace-clever-nr-gen.tsv 5-70000-replace-clever-nr-gen.txt

Result exploration

Table for expansion + generation:

Table for add + expansion + generation + other modalities:

• In red: the process does not seem to have converged
• In orange: 100% precision or coherence not achieved
• In green: close to 100% precision or coherence even if convergence is not garanteed

Below are the evolution curves: first for all operators with expansion + generation; then for add with expansion + generation + other modalities.

(drawn from data scaled every 100 measures)

Observations:

• add + clever-nr + generative (with non empty alignments) does not seem to converge.
• expansion + generation is slow to converge but end by converging, except for add.
• expansion + generation does not seem to converge towards 100% coherence for refine/addjoin/refadd
• The superimposition previously observed is not visible.

Analysis

Key points:

• Observation #1 may be (a) due to a bug, or (b) actually converges after more iterations as they are making progress and close to coherent networks.
• Observations #3 and #4 may be due to the incorrect starting size of networks (which is more incorrect as the number of agents grows).

Further experiments:

• Further perform add experiments with 150000 iterations!
• Check the accuracy of the 'add' behaviour, a bug is suspected.
• Run these tests with the correct network size (-Drealistic) See 20171231-NOOR.

Remarks:

This file can be retrieved from URL https://sake.re/20171218-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20171218-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20171218-NOOR.zip