20171231-NOOR

Mostly incorrect results because expansion did not work since commit 562f5c4b96893d30e2dc9fd59d935e013ce5219f: see 20180927-NOOR

Experiment design

Hypothesis: If the initial network contains the correct number of expected correspondences, then its results will be closer to what is obtained with empty alignment, than the standard initial network

Experimental setting: Same as 20171128-NOOR with 100000 iterations and additionally -Drealistic option

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2018-01-16 -- 2018-02-12 (The 2017-12-31 test was not sufficient)

Lazy lavender hash: ef0e80766f2b2177c5d3c1aadba6d0ca556780ef

Parameters: params.sh

Command line (script.sh):

. params.sh

for op in delete replace refine add addjoin refadd
do
   # clever-nr
  bash scripts/runexp.sh -d 5-100000-${op}-clever-nr-gen-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative -Drealistic=1
  bash scripts/runexp.sh -d 5-100000-${op}-clever-nr-gen java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative
  bash scripts/runexp.sh -d 5-100000-${op}-clever-nr-gen-empty java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative -Dstartempty

  # clever-nr plus im80
  bash scripts/runexp.sh -d 5-100000-${op}-clever-nr-im80-gen-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative -Drealistic=1
  bash scripts/runexp.sh -d 5-100000-${op}-clever-nr-im80-gen java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative
  bash scripts/runexp.sh -d 5-100000-${op}-clever-nr-im80-gen-empty java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy  -DimmediateRatio=80 -Dgenerative -Dstartempty

done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_151-b12 HotSpot (max heap 4.33G)

Raw results

Result exploration

In black, netwoks with a realistic size;

The figures reproduce those of [Euzenat 2017b] with 100000 runs, 5 agents and a realistic size of networks.

It displays precision, size and recall to be more precise.

Superimposition of the precision (left), size (centre) and recall (right) curves for operators with standard random network (red) random network with realistic size (black) and and empty ones (blue) [expansion+generative; 5 agents; 10 runs; 100000 games].

The first figure is without relaxation; the second one with a relaxation immediacy rate of 80%.

In blue, standard initial network size,
In black, empty initial network,
In red, realistic initial network size.

Observations:

The network with realistic size is always closer to the empty network than the one with the standard initial size. This seems to confirm the result of [Euzenat 2017b]
However, the curves do not strictly superimpose.

Analysis

Key points:

The initial size seems to have definitely an impact on final results
The non strict superimposition, may owe to the fact that these are not the same situations which are compared (randomization).

Further experiments:

This could be carried out by replaying exactly the same situations (games)

This file can be retrieved from URL https://sake.re/20171231-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20171231-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20171231-LOG.zip