20180305-NOOR

Experiment design

Hypothesis: The goal of this test is to check that the operators add and addjoin performs correctly. It enabled to verify that this was not the case. For that purpose, we use the new facility of Lazy lavender to redo exactly the same test. The hypotheses to be observed are:

Note: This report makes reference to non recorded experiment 20180302-NOOR.

Experimental setting: Same as the initial experiments 20140208-NOOR with:

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2018-03-05

Lazy lavender hash: fa69a4569fb2ecd7d8f24759d1ce0cd71080811a

Parameters: params.sh

Command line (script.sh):

JPATH=lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.
OPT="-DnbAgents=4 -DnbIterations=1000 -DnbRuns=10 -DreportPrecRec -Drealistic"

# Dry test only for generating the runs
# bash scripts/runexp.sh -d 4-10000-real-nothing java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=nothing -DsaveDir=expeRun -DsaveInit -DsaveParams -DsaveGames

LOADOPT="-DloadDir=expeRun -DloadEnv -DloadAgents -DreplayGames"

bash scripts/runexp.sh -d 4-10000-real-add java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=add ${LOADOPT}

bash scripts/runexp.sh -d 4-10000-real-addjoin java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=addjoin ${LOADOPT}

bash scripts/runexp.sh -d 4-10000-real-add-syntactic java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=add -Dsyntactic=true ${LOADOPT} -DsaveDir=addRes -DsaveFinal

bash scripts/runexp.sh -d 4-10000-real-addjoin-syntactic java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=addjoin -Dsyntactic=true ${LOADOPT} -DsaveDir=addjoinRes -DsaveFinal

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_131-b11 HotSpot

Note: This experiment has been set up to first generate the runs and then play them with different configuration. To strictly repeat it, it is necessary to reuse the same input and not generate a new one. Hence, the input is saved in the repository, the generating line above must be commented (including in Docker).

Raw results

4-10000-real-add-syntactic.tsv 4-10000-real-add-syntactic.txt 4-10000-real-add.tsv 4-10000-real-add.txt 4-10000-real-addjoin-syntactic.tsv 4-10000-real-addjoin-syntactic.txt 4-10000-real-addjoin.tsv 4-10000-real-addjoin.txt

Result exploration

All pictures:

We can first show how the problem was found from the data of 20180302-NOOR (hash: 65b05f404210afb600a52368cb8e1994f21dd5b8):

These results come form the same single run repeated with the different operators.

After, fixing the bug (hash: fa69a4569fb2ecd7d8f24759d1ce0cd71080811a) the results averaged on 10 runs become:

They can be described by the following table:

Table for expansion + generation:

opsuccess
rate
network
size
incoherence
degree
semantic
precision
semantic
F-measure
semantic
recall
syntactic
precision
syntactic
F-measure
syntactic
recall
maximum
convergence
add0.89460.160.800.580.450.510.360.28944
addjoin0.93460.160.800.580.450.510.360.28944

Observations: It is clear that the initial software was faultly. After correcting the bug, all expectations are met: The goal of this test is to check that the operators add and addjoin perform correctly. It enabled to verify that this was not the case. For that purpose, we use the new facility of Lazy lavender to redo exactly the same test. The hypotheses to be observed are:

It happens that, though the curves show clearly that addjoins converges faster than add, the figures report that they converges at the same game 944. This is really strange, especially when compared to what is obtained at 20180302-NOOR. This is likely due to one especially long run (convergence figure is the last game in the last run), but we do not have the data to check it.

Analysis

Key points: It seems that a bug has indeed been fixed.

Further experiments: All experiments using addjoin (and refadd) have to be rerun with the new version of Lazy lavender.

This experiment has been rerun as 20180826-NOOR


This file can be retrieved from URL https://sake.re/20180305-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180305-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr.

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180305-LOG.zip