20170530-NOOR

Experiment design

Revision of networks of ontologies with expansion, relaxation and generation starting with random network (4 agents; 10 runs; 10000 games; del/replace/refine/add/addjoin/refadd)

Hypotheses:

Generating correspondences when needed should not be different from not doing it

Experimental setting: Same as 20170529-NOOR with generation modality.

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 20170530

Lazy lavender hash: 997c4a9c1b698616c49342bb1fa5a6b8adf7faf8

Parameters: params.sh

Command line (script.sh):

for op in delete replace refine add addjoin refadd
do
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results

Result exploration

test	success rate	network size	incoherence degree	semantic precision	semantic F-measure	semantic recall	maximum convergence
4-10000-delete-clever-nr-gen	0.94	61	0.12	0.84	0.70	0.59	9099
4-10000-replace-clever-nr-gen	0.94	63	0.12	0.84	0.70	0.59	8857
4-10000-refine-clever-nr-gen	0.94	71	0.15	0.81	0.78	0.75	7063
4-10000-add-clever-nr-gen	0.91	75	0.22	0.71	0.76	0.82	9344
4-10000-addjoin-clever-nr-gen	0.95	75	0.21	0.75	0.78	0.81	6230
4-10000-refadd-clever-nr-gen	0.95	78	0.21	0.73	0.79	0.86	5840
4-10000-delete-clever-nr-im80-gen	0.94	52	0.00	1.00	0.60	0.43	8973
4-10000-replace-clever-nr-im80-gen	0.94	57	0.00	1.00	0.65	0.48	6782
4-10000-refine-clever-nr-im80-gen	0.93	63	0.00	1.00	0.77	0.62	6092
4-10000-add-clever-nr-im80-gen	0.91	68	0.00	1.00	0.78	0.64	9907
4-10000-addjoin-clever-nr-im80-gen	0.95	68	0.00	1.00	0.79	0.65	9501
4-10000-refadd-clever-nr-im80-gen	0.94	74	0.00	1.00	0.87	0.76	8495

(drawn from data scaled every 100 measures)

Analysis

Key points:

Contrary to our expectation, the generative modality systematically allows agents to achieve better results than plain operators with expansion.
It is especially better in terms of recall.
It is especially better for delete, replace, refine.
This seems to be because generation allows to test correspondences beforehand, without waiting for other correspondences to be discarded

Further experiments:

This generating modality can be tested with any other condition
It can be tested now starting with empty alignments (see 20170531-NOOR)

This file can be retrieved from URL https://sake.re/20170530-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20170530-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)