20170216-NOOR

Although, the hypothesis is not invalidated, the results for addjoin and refadd were not correct (see 20180529-NOOR)

Experiment design

Revision of networks of ontologies with expansion (4 agents; 10000 iterations)

Hypotheses:

expanding alignments when deleting correspondence will improve recall in the long run.
random/protected may not converge; clever will.
faster convergence can be obtained from random to clever-nr-comp.
the order between various operators is preserved.
precision is increased; recall preserved

Experimental setting: Repetition of 20170214a-NOOR and 20170214b-NOOR with expansion operators So all operators: delete/replace/refine/add/addjoin/refadd and modalities random/protected/clever with nonRedundancy and ontology look-up variants

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 20170215-20170216

Lazy lavender hash: dc342dca26a5a6feff895b2a1829b04e7dfaa5ae

Parameters: params.sh

Command line (script.sh):

for mod in delete replace refine add addjoin refadd
do
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${mod} -DexpandAlignments=random
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${mod} -DexpandAlignments=protected
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${mod} -DexpandAlignments=protected -DnonRedundancy
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${mod} -DexpandAlignments=clever
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${mod} -DexpandAlignments=clever -DnonRedundancy
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${mod} -DexpandAlignments=clever -DnonRedundancy -DontoLookup
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results

Result exploration

This is a lot of data.

First we look at convergence. It appears that random and protected do not seem to converge:

Convergence	random	protected	protected-nr	clever	clever-nr	clever-nr-comp
delete	9997	9998	9991	1952	1764	1778
replace	9999	9997	9995	2810	2690	2203
refine	9996	9998	9998	2498	4133	2519
add	9993	9995	9998	3534	3665	3542
addjoin	9996	9994	9994	2621	2292	2322
refadd	9985	9998	9999	4421	3405	3383

Hence from now on, we will only analyse the results of the clever modality (all data is in the files below).

To have an idea of the evolution of alignments, here is the size of the network for clever-nr on the 3000 first runs.

The figures for coherence/precision/recall/F-measure are as follows:

Coherence	clever	clever-nr	Clever-nr-comp
delete	0.05	0.05	0.01
replace	0.11	0.11	0.08
refine	0.17	0.16	0.08
add	0.21	0.21	0.22
addjoin	0.22	0.19	0.20
refadd	0.25	0.23	0.22

Precision	clever	clever-nr	Clever-nr-comp
delete	0.91	0.91	0.97
replace	0.84	0.84	0.86
refine	0.79	0.78	0.88
add	0.72	0.73	0.71
addjoin	0.72	0.75	0.74
refadd	0.69	0.71	0.72

Recall	clever	clever-nr	Clever-nr-comp
delete	0.22	0.22	0.16
replace	0.39	0.38	0.30
refine	0.56	0.55	0.41
add	0.63	0.66	0.68
addjoin	0.63	0.59	0.57
refadd	0.78	0.76	0.73

F-measure	clever	clever-nr	Clever-nr-comp
delete	0.35	0.35	0.27
replace	0.53	0.52	0.44
refine	0.66	0.65	0.56
add	0.67	0.69	0.69
addjoin	0.67	0.66	0.64
refadd	0.73	0.73	0.72

F-measure evolution on clever (solid), clever-nr (dashed) and clever-nr-com (dotted), first 5000 games:

(drawn from data scaled every 100 measures)

Analysis

Key points:

Operators without memory do not converge
Operators with memory provide a large increase on recall and F-measure, at the expense of precision
The order between operators is globally preserved; but the add/addjoin still strange
Precision increases with nr and comp; it is unclear for coherence; recall decreases
In the end nr makes F-measure increase; comp makes it decrease!

Further experiments:

sortout the add/addjoin discrepancy (see 20180529-NOOR)
test if ever the first three modalities would eventually converge... (see 20170222-NOOR)

This file can be retrieved from URL https://sake.re/20170216-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20170216-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)