20180529-NOOR

Experiment design

Designer: Jérôme Euzenat (INRIA) (2017-02-16)

Revision of networks of ontologies with expansion (4 agents; 10 runs; 10000 games; delete/replace/refine/add/addjoin/refadd)

Hypotheses:

expanding alignments when deleting correspondence will improve recall in the long run.
random/protected may not converge; clever will.
faster convergence can be obtained from random to clever-nr-comp.
the order between various operators is preserved.
precision is increased; recall preserved

Experimental setting: Same as 20170216-NOOR replaying the same runs 20180308-NOOR (putatively) and after fixing addjoin and expansion.

Experiment

Experimenter: Jérôme Euzenat (INRIA) (2018-05-29)

Date: 2018-05-29

Lazy lavender hash: d50e70f87bca76951ec2f149dc8ae1d42b9a1a28

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameters: params.sh

Command line (script.sh):

#!/bin/bash

JPATH=lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OPT="-DnbAgents=4 -DnbIterations=10000 -DnbRuns=10 -DreportPrecRec"

LOADOPT="-DloadDir=input/expeRun -DloadEnv -DloadAgents -DreplayGames"

for op in delete replace refine add addjoin refadd
do
   bash scripts/runexp.sh -d 4-10000-${op}-random java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=random
   bash scripts/runexp.sh -d 4-10000-${op}-protected java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=protected
   bash scripts/runexp.sh -d 4-10000-${op}-protected-nr java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=protected -DnonRedundancy
   bash scripts/runexp.sh -d 4-10000-${op}-clever java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever
   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy

done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun.zip Then unzip expeRun.zip -d input

Raw results

4-10000-add-clever-nr.tsv 4-10000-add-clever-nr.txt 4-10000-add-clever.tsv 4-10000-add-clever.txt 4-10000-add-protected-nr.tsv 4-10000-add-protected-nr.txt 4-10000-add-protected.tsv 4-10000-add-protected.txt 4-10000-add-random.tsv 4-10000-add-random.txt 4-10000-addjoin-clever-nr.tsv 4-10000-addjoin-clever-nr.txt 4-10000-addjoin-clever.tsv 4-10000-addjoin-clever.txt 4-10000-addjoin-protected-nr.tsv 4-10000-addjoin-protected-nr.txt 4-10000-addjoin-protected.tsv 4-10000-addjoin-protected.txt 4-10000-addjoin-random.tsv 4-10000-addjoin-random.txt 4-10000-delete-clever-nr.tsv 4-10000-delete-clever-nr.txt 4-10000-delete-clever.tsv 4-10000-delete-clever.txt 4-10000-delete-protected-nr.tsv 4-10000-delete-protected-nr.txt 4-10000-delete-protected.tsv 4-10000-delete-protected.txt 4-10000-delete-random.tsv 4-10000-delete-random.txt 4-10000-refadd-clever-nr.tsv 4-10000-refadd-clever-nr.txt 4-10000-refadd-clever.tsv 4-10000-refadd-clever.txt 4-10000-refadd-protected-nr.tsv 4-10000-refadd-protected-nr.txt 4-10000-refadd-protected.tsv 4-10000-refadd-protected.txt 4-10000-refadd-random.tsv 4-10000-refadd-random.txt 4-10000-refine-clever-nr.tsv 4-10000-refine-clever-nr.txt 4-10000-refine-clever.tsv 4-10000-refine-clever.txt 4-10000-refine-protected-nr.tsv 4-10000-refine-protected-nr.txt 4-10000-refine-protected.tsv 4-10000-refine-protected.txt 4-10000-refine-random.tsv 4-10000-refine-random.txt 4-10000-replace-clever-nr.tsv 4-10000-replace-clever-nr.txt 4-10000-replace-clever.tsv 4-10000-replace-clever.txt 4-10000-replace-protected-nr.tsv 4-10000-replace-protected-nr.txt 4-10000-replace-protected.tsv 4-10000-replace-protected.txt 4-10000-replace-random.tsv 4-10000-replace-random.txt

Result exploration

As observed in 20170216-NOOR, purely random addition leads to success rate stabilizing around 80% (non converging as seen in 20170222-NOOR):

The size of the network for random/protected tends to get very high (around 100) and the same for all operators (but for delete and addjoin):

The size for clever (and clever-nr) is lower, according to operator preservation order and growing towards that of the reference alignment:

The difference in F-measure for clever and clever with non redundancy is non oriented (this also holds for precision and recall):

Finally, clever-nr improves on recall and F-measure, but decreases precision with respect to non expansion (20180308-NOOR):

test	success rate	network size	incoherence degree	semantic precision	semantic F-measure	semantic recall	maximum convergence
4-10000-add-clever-nr	0.96	59	0.25	0.70	0.69	0.69	3281
4-10000-add-clever	0.96	60	0.25	0.70	0.69	0.68	3709
4-10000-add-protected-nr	0.79	101	0.31	0.47	0.60	0.83	8985
4-10000-add-protected	0.79	101	0.32	0.46	0.59	0.82	8136
4-10000-add-random	0.79	101	0.32	0.47	0.60	0.82	3608
4-10000-addjoin-clever-nr	0.98	58	0.22	0.72	0.68	0.65	1856
4-10000-addjoin-clever	0.98	59	0.24	0.70	0.68	0.65	3393
4-10000-addjoin-protected-nr	0.79	100	0.34	0.46	0.59	0.82	8555
4-10000-addjoin-protected	0.79	101	0.33	0.46	0.60	0.83	8278
4-10000-addjoin-random	0.79	101	0.33	0.47	0.60	0.83	9731
4-10000-delete-clever-nr	0.98	15	0.01	0.98	0.26	0.15	2473
4-10000-delete-clever	0.98	13	0.01	0.98	0.25	0.14	1588
4-10000-delete-protected-nr	0.82	54	0.20	0.55	0.50	0.45	5010
4-10000-delete-protected	0.83	54	0.18	0.57	0.49	0.44	4031
4-10000-delete-random	0.82	54	0.20	0.56	0.49	0.44	5247
4-10000-refadd-clever-nr	0.96	70	0.23	0.69	0.73	0.78	4174
4-10000-refadd-clever	0.97	72	0.24	0.70	0.75	0.82	3708
4-10000-refadd-protected-nr	0.79	117	0.34	0.44	0.60	0.93	9985
4-10000-refadd-protected	0.79	119	0.34	0.45	0.60	0.90	9991
4-10000-refadd-random	0.79	118	0.33	0.44	0.60	0.91	10000
4-10000-refine-clever-nr	0.97	38	0.10	0.87	0.59	0.44	3971
4-10000-refine-clever	0.97	40	0.10	0.85	0.60	0.47	2327
4-10000-refine-protected-nr	0.76	108	0.32	0.47	0.61	0.85	9978
4-10000-refine-protected	0.76	108	0.32	0.47	0.61	0.85	9967
4-10000-refine-random	0.77	108	0.33	0.47	0.60	0.86	9808
4-10000-replace-clever-nr	0.97	28	0.07	0.89	0.43	0.29	1829
4-10000-replace-clever	0.97	27	0.07	0.87	0.43	0.29	1661
4-10000-replace-protected-nr	0.76	101	0.32	0.47	0.59	0.80	7271
4-10000-replace-protected	0.76	101	0.33	0.47	0.60	0.83	7925
4-10000-replace-random	0.76	100	0.32	0.48	0.60	0.82	7939

Analysis

Analyst: Jérôme Euzenat (INRIA) (2018-08-16)

Key points:

All hypotheses hold, but
The difference between clever and clever-nr may be attributed to the randomness of added correspondences
As observed in 20170216-NOOR, recall is not preserved but overall F-measure is always improved.

Further experiments: none

This file can be retrieved from URL https://sake.re/20180529-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180529-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180529-NOOR.zip