20190712-NOOR

2019-07-12: This experiment does not allow to conclude

Experiment design

Designer: Jérôme Euzenat (INRIA) ()

Revision of networks of ontologies with expansion, relaxation and generation starting with empty network (4 agents; 10 runs; 20000 games; delete/replace/refine/add/addjoin/refadd)

Hypotheses: Starting from scratch and generating correspondences when needed should not be different from starting with initial alignments

Experimental setting: Same as 20180827-NOOR replaying the 20000 runs of 20190604-NOOR after major code revision

Experiment

Experimenter: Jérôme Euzenat (INRIA) (2019-07-12)

Date: 2019-07-12

Lazy lavender hash: abd46811f38e9c765b2980cbb2fe504aff7b3e5e

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Variation of: 20180827-NOOR

Parameters: params.sh

Command line (script.sh):

. params.sh

OUTPUT=${OUTPUT}${LABEL}

for op in ${OPS}
do
   bash scripts/runexp.sh -p ${OUTPUT} -d ${DIRPREF}-${op}-clever-nr${postfix} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op}
   bash scripts/runexp.sh -p ${OUTPUT} -d ${DIRPREF}-${op}-clever-nr-im80${postfix} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DimmediateRatio=80
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: 24 * Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz with 20GB RAM / Linux 4.15.17-1-pve / Java(TM) SE Runtime Environment 1.8.0_151 with 4.33G max heap size

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun2.zip Then unzip expeRun2.zip -d input

Raw results

4-20000-add-clever-nr-gen-empty.tsv 4-20000-add-clever-nr-gen-empty.txt 4-20000-add-clever-nr-im80-gen-empty.tsv 4-20000-add-clever-nr-im80-gen-empty.txt 4-20000-addjoin-clever-nr-gen-empty.tsv 4-20000-addjoin-clever-nr-gen-empty.txt 4-20000-addjoin-clever-nr-im80-gen-empty.tsv 4-20000-addjoin-clever-nr-im80-gen-empty.txt 4-20000-delete-clever-nr-gen-empty.tsv 4-20000-delete-clever-nr-gen-empty.txt 4-20000-delete-clever-nr-im80-gen-empty.tsv 4-20000-delete-clever-nr-im80-gen-empty.txt 4-20000-refadd-clever-nr-gen-empty.tsv 4-20000-refadd-clever-nr-gen-empty.txt 4-20000-refadd-clever-nr-im80-gen-empty.tsv 4-20000-refadd-clever-nr-im80-gen-empty.txt 4-20000-refine-clever-nr-gen-empty.tsv 4-20000-refine-clever-nr-gen-empty.txt 4-20000-refine-clever-nr-im80-gen-empty.tsv 4-20000-refine-clever-nr-im80-gen-empty.txt 4-20000-replace-clever-nr-gen-empty.tsv 4-20000-replace-clever-nr-gen-empty.txt 4-20000-replace-clever-nr-im80-gen-empty.tsv 4-20000-replace-clever-nr-im80-gen-empty.txt

Result exploration

Results are compared with the previous ones (20180827-NOOR, dashed on 10000) and those starting with random alignments (20190624-NOOR, dotted).

test	success rate	network size	incoherence degree	semantic precision	semantic F-measure	semantic recall	maximum convergence
4-20000-delete-clever-nr-gen-empty	0.96	58	0.14	0.82	0.63	0.51	11462
4-20000-replace-clever-nr-gen-empty	0.96	61	0.17	0.79	0.66	0.57	11872
4-20000-refine-clever-nr-gen-empty	0.96	79	0.21	0.76	0.75	0.75	11262
4-20000-add-clever-nr-gen-empty	0.95	82	0.15	0.82	0.78	0.74	16516
4-20000-addjoin-clever-nr-gen-empty	0.97	80	0.16	0.82	0.78	0.74	12754
4-20000-refadd-clever-nr-gen-empty	0.96	80	0.15	0.80	0.77	0.75	16814
4-20000-delete-clever-nr-im80-gen-empty	0.96	50	0.00	1.00	0.54	0.37	10826
4-20000-replace-clever-nr-im80-gen-empty	0.96	50	0.00	1.00	0.56	0.39	12595
4-20000-refine-clever-nr-im80-gen-empty	0.96	70	0.00	1.00	0.77	0.63	14890
4-20000-add-clever-nr-im80-gen-empty	0.95	76	0.00	1.00	0.76	0.62	17246
4-20000-addjoin-clever-nr-im80-gen-empty	0.97	75	0.00	1.00	0.77	0.62	14518
4-20000-refadd-clever-nr-im80-gen-empty	0.96	80	0.00	1.00	0.81	0.68	11469

Analysis

Analyst: Jérôme Euzenat (INRIA) (2019-08-29)

Key points:

Convergence seems to be established at 20000
The size is higher than starting randomly, both with and without relaxation
Precision is higher without relaxation, this is explained because starting empty, correspondences are immediately tested and are not easy to shadow; with relaxation precision reaches 100%
Recall seems lower
In consequence, F-measure is roughly the same without relaxation (but this is by compensation) and it is lower with relaxation (due to higher recall)
This contradicts the hypothesis

Further experiments:

These experiments should be redone with strengthening to check if it is related to the same kind of issues

This file can be retrieved from URL https://sake.re/20190712-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20190712-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20190712-NOOR.zip