20180305-NOOR

Experiment design

Hypothesis: The goal of this test is to check that the operators add and addjoin performs correctly. It enabled to verify that this was not the case. For that purpose, we use the new facility of Lazy lavender to redo exactly the same test. The hypotheses to be observed are:

They have the same size at convergence
Addjoin converges faster
Addjoin has a higher semantic precision all the time (idem for syntactic)
Addjoin has a lower semantic recall all the time (syntactic are equal)

Note: This report makes reference to non recorded experiment 20180302-NOOR.

Experimental setting: Same as the initial experiments 20140208-NOOR with:

repeatable series of events
precision and recall collected semantically and syntactically

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2018-03-05

Lazy lavender hash: fa69a4569fb2ecd7d8f24759d1ce0cd71080811a

Parameters: params.sh

Command line (script.sh):

JPATH=lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.
OPT="-DnbAgents=4 -DnbIterations=1000 -DnbRuns=10 -DreportPrecRec -Drealistic"

# Dry test only for generating the runs
# bash scripts/runexp.sh -d 4-1000-nothing-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=nothing -DsaveDir=expeRun -DsaveInit -DsaveParams -DsaveGames

LOADOPT="-DloadDir=expeRun -DloadEnv -DloadAgents -DreplayGames"

bash scripts/runexp.sh -d 4-1000-add-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=add ${LOADOPT}

bash scripts/runexp.sh -d 4-1000-addjoin-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=addjoin ${LOADOPT}

bash scripts/runexp.sh -d 4-1000-add-syntactic-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=add -Dsyntactic=true ${LOADOPT} -DsaveDir=addRes -DsaveFinal

bash scripts/runexp.sh -d 4-1000-addjoin-syntactic-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=addjoin -Dsyntactic=true ${LOADOPT} -DsaveDir=addjoinRes -DsaveFinal

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_131-b11 HotSpot

Note: This experiment has been set up to first generate the runs and then play them with different configuration. To strictly repeat it, it is necessary to reuse the same input and not generate a new one. Hence, the input is saved in the repository, the generating line above must be commented (including in Docker).

Raw results

4-1000-add-syntactic-real.tsv 4-1000-add-syntactic-real.txt 4-1000-add-real.tsv 4-1000-add-real.txt 4-1000-addjoin-syntactic-real.tsv 4-1000-addjoin-syntactic-real.txt 4-1000-addjoin-real.tsv 4-1000-addjoin-real.txt

Result exploration

All pictures:

red: measure for add (dashed: syntactic)
blue: measure for addjoin (dashed: syntactic)

We can first show how the problem was found from the data of 20180302-NOOR (hash: 65b05f404210afb600a52368cb8e1994f21dd5b8):

These results come form the same single run repeated with the different operators.

After, fixing the bug (hash: fa69a4569fb2ecd7d8f24759d1ce0cd71080811a) the results averaged on 10 runs become:

They can be described by the following table:

Table for expansion + generation:

op	success rate	network size	incoherence degree	semantic precision	semantic F-measure	semantic recall	syntactic precision	syntactic F-measure	syntactic recall	maximum convergence
add	0.89	46	0.16	0.80	0.58	0.45	0.51	0.36	0.28	944
addjoin	0.93	46	0.16	0.80	0.58	0.45	0.51	0.36	0.28	944

Observations: It is clear that the initial software was faultly. After correcting the bug, all expectations are met: The goal of this test is to check that the operators add and addjoin perform correctly. It enabled to verify that this was not the case. For that purpose, we use the new facility of Lazy lavender to redo exactly the same test. The hypotheses to be observed are:

Networks have the same size at convergence
Addjoin converges faster
Addjoin has a higher semantic precision all the time (idem for syntactic)
Addjoin has a lower semantic recall all the time (syntactic are equal) Only the last statement in case of syntactic seems slightly problematic and this requires further investigation.

It happens that, though the curves show clearly that addjoins converges faster than add, the figures report that they converges at the same game 944. This is really strange, especially when compared to what is obtained at 20180302-NOOR. This is likely due to one especially long run (convergence figure is the last game in the last run), but we do not have the data to check it.

Analysis

Key points: It seems that a bug has indeed been fixed.

Further experiments: All experiments using addjoin (and refadd) have to be rerun with the new version of Lazy lavender.

This experiment has been rerun as 20180826-NOOR

This file can be retrieved from URL https://sake.re/20180305-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180305-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180305-LOG.zip