20170214b-NOOR

Results for addjoin and refadd are underestimated (see 20180308-NOOR)

Experiment design

Revision of networks of ontologies comparing refine/addjoin/refadd new operators (4 agents; 10000 iterations)

Hypotheses:

H1: Addjoin is faster to converge than add and achieves the same results
H2: refine is better than replace (in 20170208-NOOR for instance)
H3: refadd is better than add and refine (especially with better recall)

Experimental setting: Exactly the same as 20170209-NOOR with improved software (corrected bugs)

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 20170214-20170215

Lazy lavender hash: dc342dca26a5a6feff895b2a1829b04e7dfaa5ae

Parameters: params.sh

Command line (script.sh):

java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=refine

java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=addjoin

java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=refadd

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results

20170214b-NOOR-4-10000-addjoin.tsv 20170214b-NOOR-4-10000-addjoin.txt 20170214b-NOOR-4-10000-refadd.tsv 20170214b-NOOR-4-10000-refadd.txt 20170214b-NOOR-4-10000-refine.tsv 20170214b-NOOR-4-10000-refine.txt

Result exploration

modality	size	success	convergence	Incoherence degree	Precision	F-measure	Recall
refine	20	.99	1133	.02	.96	.37	.23
addjoin	23	.99	1004	.10	.84	.43	.29
refadd	41	.99	1266	.09	.86	.62	.48

In red the average success rate (over 10 runs) of refine;
In blue the average success rate (over 10 runs) of addjoin;
In green the average success rate (over 10 runs) of refadd;
In dashed, the semantic F-measure.

For dealing with hypothesis H1, we compare with the result of add on 20170214a-NOOR:

In blue the average success rate (over 10 runs) of addjoin;
In red the average success rate (over 10 runs) of add;
In dashed, the semantic F-measure.

For dealing with hypothesis H2, we compare with the result of replace on 20170214a-NOOR:

In blue the average success rate (over 10 runs) of refine;
In red the average success rate (over 10 runs) of replace;
In dashed, the semantic F-measure.

Above: semantic precision;
Below: semantic recall.

(drawn from data scaled every 100 measures)

Analysis

Key points:

Concerning H1, it seems not possible to conclude: addjoin seems indeed to converge faster on average while on the table, add has its maximum convergence step at 1235 instead of 1523 fo addjoin. Moreover, they achieve significant F-measure difference, so it would be better to compare them on the exact same scenarii.
Concerning H2, refine seems slower and achieve an higher F-measure than replace
With respect to H3, refadd is better in F-measure than refine and addjoin although it is slower to converge. It also, as expected, has better recall.

Further experiments:

Test H1 with the exact same scenario.

This file can be retrieved from URL https://sake.re/20170214b-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20170214b-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)