20170215a-NOOR

Results for addjoin and refadd are underestimated (see 20180311-NOOR)

Experiment design

Revision of networks of ontologies with immediate consumption probability (4 agents; 10000 iterations)

Hypothesis:

introducing the capability for agent to sometimes ignore the most specific applicable correspondences (with given immediate consumption probability) will improve precision by fighting shadowing.

Experimental setting: Repetition of 20170208-NOOR and 20170209-NOOR with consuption probability set to 80%

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 20170215

Lazy lavender hash: dc342dca26a5a6feff895b2a1829b04e7dfaa5ae

Parameters: params.sh

Command line (script.sh):

for mod in delete replace refine add addjoin refadd
do
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.1.9.jar:lib/slf4j/logback-core-1.1.9.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec -DrevisionModality=${mod} -DimmediateRatio=80
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results

20170215a-NOOR-4-10000-add-im80.tsv 20170215a-NOOR-4-10000-add-im80.txt 20170215a-NOOR-4-10000-addjoin-im80.tsv 20170215a-NOOR-4-10000-addjoin-im80.txt 20170215a-NOOR-4-10000-delete-im80.tsv 20170215a-NOOR-4-10000-delete-im80.txt 20170215a-NOOR-4-10000-refadd-im80.tsv 20170215a-NOOR-4-10000-refadd-im80.txt 20170215a-NOOR-4-10000-refine-im80.tsv 20170215a-NOOR-4-10000-refine-im80.txt 20170215a-NOOR-4-10000-replace-im80.tsv 20170215a-NOOR-4-10000-replace-im80.txt

Result exploration

With the standard modalities:

modality	success	size	convergence	Precision	F-measure	Recall
delete	0.99520004	6.0	622	1.0	0.13	0.06976745
replace	0.99095	11.5	947	1.0	0.20	0.115116276
refine	0.98639995	19.0	1708	1.0	0.33	0.20232558
add	0.98098993	25.4	2159	1.0	0.37	0.22906978
addjoin	0.99003	19.3	824	1.0	0.33	0.19767441
refadd	0.9847599	40.2	1743	1.0	0.61	0.4360465

In plain, average success rate (over 10 runs);
In dashed, the semantic F-measure.

Compared to raw operators:

In plain, raw operator semantic F-measure;
In dashed, 80% immediate consumption probability semantic F-measure.

(drawn from data scaled every 100 measures)

Analysis

Key points:

The results converge in all cases to 100% precision and 0% incoherence.
The behaviour of addjoin as related to add is still unclear
Recall and F-measure are however lower

Further experiments:

sortout the add/addjoin discrepancy
associate it with the expansion operator (see 20170215b-NOOR)

This file can be retrieved from URL https://sake.re/20170215a-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20170215a-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)