20191218-POPR

Experiment design

Designer: Jérôme Euzenat (2019-12-18)

Revision of networks of ontologies with population and synchronization (4 populations * 4 agents; 10 runs; 50000 games; delete/replace/refine/add/addjoin/refadd)

Hypotheses: Even with high frequency synchronisation, populations achieve the same quality as no synchronisation

Experimental setting: Variation of 20190701-POPR with a larger number of games to test convergence

Experiment

Experimenter: Jérôme Euzenat (INRIA) (2019-12-18)

Date: 2019-12-18

Lazy lavender hash: ff04e315c7a5912831407320a070a776250c91c2

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Variation of: 20190701-NOOR

Parameters: params.sh

Command line (script.sh):

. params.sh

OUTPUT=${OUTPUT}${LABEL}

for op in ${OPS}
do
   bash scripts/runexp.sh -p ${OUTPUT} -d ${DIRPREF}-${op}-sncmaj0 java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -Dexperiment=fr.inria.exmo.lazylavender.expe.PopulatedARExperiment ${OPT} ${LOADOPT} -DrevisionModality=${op} -DsynchronisationMode=majority -DsynchronizationRate=0
   bash scripts/runexp.sh -p ${OUTPUT} -d ${DIRPREF}-${op}-sncmaj${SYNC} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -Dexperiment=fr.inria.exmo.lazylavender.expe.PopulatedARExperiment ${OPT} ${LOADOPT} -DrevisionModality=${op} -DsynchronisationMode=majority -DsynchronizationRate=${SYNCRATE}
   bash scripts/runexp.sh -p ${OUTPUT} -d ${DIRPREF}-${op}-sncgen${SYNC} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -Dexperiment=fr.inria.exmo.lazylavender.expe.PopulatedARExperiment ${OPT} ${LOADOPT} -DrevisionModality=${op} -DsynchronisationMode=general -DsynchronizationRate=${SYNCRATE}
   bash scripts/runexp.sh -p ${OUTPUT} -d ${DIRPREF}-${op}-sncspc${SYNC} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -Dexperiment=fr.inria.exmo.lazylavender.expe.PopulatedARExperiment ${OPT} ${LOADOPT} -DrevisionModality=${op} -DsynchronisationMode=specific -DsynchronizationRate=${SYNCRATE}
done

Class used: Population, PopulatedARExperiment, PopulationAlignmentAdjustingAgent, NOOEnvironment, ActionLogger, AverageLogger, Monitor

Execution environment: 24 * Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz with 20GB RAM / Linux 4.15.17-1-pve / Java(TM) SE Runtime Environment 1.8.0_151 with 4.33G max heap size

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expePop50k.zip Then unzip expePop50k.zip -d input

Raw results

4x4-50000-add-sncgen100.tsv 4x4-50000-add-sncgen100.txt 4x4-50000-add-sncmaj0.tsv 4x4-50000-add-sncmaj0.txt 4x4-50000-add-sncmaj100.tsv 4x4-50000-add-sncmaj100.txt 4x4-50000-add-sncspc100.tsv 4x4-50000-add-sncspc100.txt 4x4-50000-addjoin-sncgen100.tsv 4x4-50000-addjoin-sncgen100.txt 4x4-50000-addjoin-sncmaj0.tsv 4x4-50000-addjoin-sncmaj0.txt 4x4-50000-addjoin-sncmaj100.tsv 4x4-50000-addjoin-sncmaj100.txt 4x4-50000-addjoin-sncspc100.tsv 4x4-50000-addjoin-sncspc100.txt 4x4-50000-delete-sncgen100.tsv 4x4-50000-delete-sncgen100.txt 4x4-50000-delete-sncmaj0.tsv 4x4-50000-delete-sncmaj0.txt 4x4-50000-delete-sncmaj100.tsv 4x4-50000-delete-sncmaj100.txt 4x4-50000-delete-sncspc100.tsv 4x4-50000-delete-sncspc100.txt 4x4-50000-refadd-sncgen100.tsv 4x4-50000-refadd-sncgen100.txt 4x4-50000-refadd-sncmaj0.tsv 4x4-50000-refadd-sncmaj0.txt 4x4-50000-refadd-sncmaj100.tsv 4x4-50000-refadd-sncmaj100.txt 4x4-50000-refadd-sncspc100.tsv 4x4-50000-refadd-sncspc100.txt 4x4-50000-refine-sncgen100.tsv 4x4-50000-refine-sncgen100.txt 4x4-50000-refine-sncmaj0.tsv 4x4-50000-refine-sncmaj0.txt 4x4-50000-refine-sncmaj100.tsv 4x4-50000-refine-sncmaj100.txt 4x4-50000-refine-sncspc100.tsv 4x4-50000-refine-sncspc100.txt 4x4-50000-replace-sncgen100.tsv 4x4-50000-replace-sncgen100.txt 4x4-50000-replace-sncmaj0.tsv 4x4-50000-replace-sncmaj0.txt 4x4-50000-replace-sncmaj100.tsv 4x4-50000-replace-sncmaj100.txt 4x4-50000-replace-sncspc100.tsv 4x4-50000-replace-sncspc100.txt

Result exploration

It is noteworthy that the curves draw one point every 25 games and that syncronisation occurs every 100 games... using the usual pattern of one point every 100 games give an impression of smoothness to the results which is not correct.

Observations:

• The level of quality with synchronisation meets that of no synchronisation:
  • maj100 and spc100 provide a F-measure equal or 1 point superior
  • gen100, beside delete and replace, is lower.
• refadd and refine do not seem to converge when synchronisation occurs, especially for most general and most specific. More precisely,
  • Precision: values very unstable for gen and spc; most general does not seem to converge, most specific seems to converge albeit more slowly than majority.
  • Recall: converges quickly (but remains unstable for gen, albeit at a higher level than the others

Analysis

Analyst: Jérôme Euzenat (INRIA) (2019-12-22)

Key points:

• The hypothesis does not seem to hold for the most general mode, but this may be due to lack of convergence
• The question of why there is still so much size variation is open...

Conclusions:

• The hypothesis is supported for the majority and most specific synchronisation approach but does not seems true of the most general one

Further experiments:

• Could be run with 200/500 synchronisation rate to better assess the asymptotic behaviour, instead of larger number of games
• It would be good to explain the behaviour of refine and refadd (obviously due to the refine part)

This file can be retrieved from URL https://sake.re/20191218-POPR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20191218-POPR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20191218-POPR.zip