20180307-NOOR

Experiment design

Hypothesis: Realistic numbers of correspondences in initial alignments provide better final results

Experimental setting: Same as 20180308-NOOR with realistic numbers of correspondences in each initial alignment

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2018-03-07

Lazy lavender hash: fa69a4569fb2ecd7d8f24759d1ce0cd71080811a

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

Parameters: params.sh

Command line (script.sh):

. params.sh

# Dry test only for generating the runs
# bash scripts/runexp.sh -d 4-5000-nothing-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=nothing -DsaveDir=input/expeRun -DsaveInit -DsaveParams -DsaveGames

for mod in ${OPS}
do
   bash scripts/runexp.sh -d 4-5000-${mod}-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${mod}
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_151-b12 HotSpot

Note: This experiment has been set up to first generate the runs and then play them with different configuration. To strictly repeat it, it is necessary to reuse the same input and not generate a new one. Hence, the input is saved in the repository, the generating line above must be commented (including in Docker).

Result exploration

op	success rate	network size	semantic precision	semantic F-measure	semantic recall	maximum convergence
delete	0.99	24	0.91	0.30	0.18	743
replace	0.99	24	0.91	0.30	0.18	743
refine	0.98	33	0.92	0.45	0.29	1171
add	0.98	43	0.79	0.55	0.42	1191
addjoin	0.99	43	0.79	0.55	0.42	1048
refadd	0.98	59	0.80	0.69	0.60	1048

In plain: success rate;
In dashed: F-measure.

In plain: precision;
In dashed: recall.

Observations:

The 0.0 in incoherence is due to a computing mistake
As now standard, add and addjoin achieve the same results. This time addjoin converges faster than add for the worst time.
delete and replace have the same results because of the way realistic correspondences are: they are always < or > hence there is never any replace to do.

Analysis

Key points: Compared with the results of 20180308-NOOR starting with realistic:

The size of obtained alignments is higher
The recall is higher, precicision not necessarily lower (may be due to randomization)
Overall F-measure is higher
The effect is far more important on delete/refine than on add/addjoin/refadd

Further experiments:

Redo this experiments with exactly the same events as 20180308-NOOR (see 20180531-NOOR)

Remarks:

This file can be retrieved from URL https://sake.re/20180307-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180307-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180307-LOG.zip