20190701-POPR

Experiment design

Designer: Fatima Danash (INRIA) (2019-06-01)

Revision of networks of ontologies with population and synchronization using majority mode (4 pops * 4 agents; 10 runs; 20000 games; delete/replace/refine/add/addjoin/refadd)

Hypotheses:

(H1) The majority synchronization mode converges faster than no synchronisation
(H2) The majority synchronisation mode provides higher F-measure than no synchronisation

Experimental setting: New population-based synchronisation

Experiment

Experimenter: Jérôme Euzenat (INRIA) (2019-07-15)

Date: 2019-07-15

Lazy lavender hash: f91a6a57e385e2bfed96840d2b6cc76893a935b0

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameters: params.sh

Command line (script.sh):

. params.sh

OUTPUT=${OUTPUT}${LABEL}

for rate in ${SYNCRATES}
do
   for op in ${OPS}
   do
    bash scripts/runexp.sh -p ${OUTPUT} -d ${DIRPREF}-${op}${postfix}${rate} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -Dexperiment=fr.inria.exmo.lazylavender.expe.PopulatedARExperiment ${OPT} ${LOADOPT} -DrevisionModality=${op} -DsynchronizationRate=${rate}
   done
done

Class used: Population, PopulatedARExperiment, PopulationAlignmentAdjustingAgent, NOOEnvironment, ActionLogger, AverageLogger, Monitor

Execution environment: 24 * Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz with 20GB RAM / Linux 4.15.17-1-pve / Java(TM) SE Runtime Environment 1.8.0_151 with 4.33G max heap size

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expePop.zip Then unzip expePop.zip -d input

Raw results

Result exploration

test	success rate	network size	incoherence degree	semantic precision	semantic F-measure	semantic recall	maximum convergence
No synchronisation (sncmaj0)
delete	0.99	12	0.00	0.99	0.58	0.42	4096
replace	0.99	12	0.00	0.99	0.58	0.42	4096
refine	0.98	18	0.01	0.99	0.63	0.47	5732
add	0.98	18	0.02	0.96	0.62	0.46	5656
addjoin	0.99	18	0.02	0.96	0.62	0.46	4096
refadd	0.98	35	0.04	0.94	0.73	0.60	9200
Majority synchronisation each 1000 games (sncmaj1000)
delete	0.99	12	0.00	0.99	0.59	0.42	4096
replace	0.99	12	0.00	0.99	0.59	0.42	4096
refine	0.98	19	0.01	0.99	0.64	0.47	5732
add	0.98	19	0.02	0.96	0.63	0.47	5656
addjoin	0.99	19	0.02	0.96	0.63	0.47	4096
refadd	0.98	37	0.04	0.94	0.74	0.61	9200
Majority synchronisation each 500 games (sncmaj500)
delete	0.99	12	0.00	0.99	0.59	0.42	4135
replace	0.99	12	0.00	0.99	0.59	0.42	4135
refine	0.98	19	0.01	0.99	0.64	0.48	7070
add	0.98	19	0.02	0.96	0.63	0.47	6278
addjoin	0.99	19	0.02	0.96	0.63	0.47	4806
refadd	0.98	37	0.03	0.94	0.74	0.61	9200
Majority synchronisation each 100 games (sncmaj100)
delete	0.99	13	0.01	0.97	0.58	0.42	14097
replace	0.99	13	0.01	0.97	0.58	0.42	14097
refine	0.97	20	0.01	0.95	0.64	0.48	19996
add	0.97	20	0.03	0.94	0.63	0.47	19817
addjoin	0.98	20	0.02	0.94	0.63	0.47	19745
refadd	0.97	37	0.04	0.91	0.74	0.62	19979

Observations:

the 100 synchronisation rate has not necessarily converged after 20000 games
The hierarchy between operators remain the same as in other experiments

Analysis

Analyst: Jérôme Euzenat (INRIA) (2019-12-09 -- 17)

Key points:

The size of the result is roughly the same (between 0 and +2 units)
The precision is the same at convergence (1000, 500)
The recall is roughly the same at convergence (between 0 and +2 points)
The more synchronisation occurs, the slower convergence is

Conclusions:

H1 is clearly not supported: this is due to the
H2 is minimally observed

Further experiments:

It may be worth testing the evolution of convergence between 500 and 0 synchronisation rate
Experiment with further synchronisation modes and strategies

This file can be retrieved from URL https://sake.re/20190701-POPR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20190701-POPR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20190701-POPR.zip