Experiment 20180308-NOOR¶

refadd > add = addjoin > refine > replace > delete

Experiment design¶

Date: 2018-03-08

Hypotheses: Adaptation operators (but add and addjoin) are ordered by F-measure

Variation of: 20170214b-NOOR

4 agents; 10 runs; 10000 games

Adaptation operators: delete replace refine add addjoin refadd

Experimental setting: Same as the initial experiments [20170214a-NOOR] and [20170214b-NOOR] after fixing errors in the implementation of addjoin (affects addjoin and refadd), with comparison on the exact same set of games (replayGames)

Variables¶

controled variables: ['revisionModality']

dependent variables: ['srate', 'size', 'inc', 'prec', 'fmeas', 'rec', 'conv']

Experiment¶

Date: 2018-03-08

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: fa69a4569fb2ecd7d8f24759d1ce0cd71080811a

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash
# Script to run experiments in bactch on a remore machine
# Procedure:
# - zip the current directory: cd ..; zip -rq lazylav.zip lazylav
# - transfer it on the execution machine: scp lazylav.zip seals-4.inrialpes.fr:/scratch/LazyLav/
# - unzip on this machine: unzip lazylav.zip; cd lazylav
# - run it there: nohup bash myscript.sh < /dev/null 2> /dev/null > /dev/null &
# - results are found the directories passed as -d

# Dry test only for generating the runs
# bash scripts/runexp.sh -d ${DIRPREF}-nothing java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} -DrevisionModality=nothing -DsaveDir=input/expeRun -DsaveInit -DsaveParams -DsaveGames

for mod in ${OPS}
do
   bash scripts/runexp.sh -d ${DIRPREF}-${mod} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${mod}

done

exit

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_151-b12 HotSpot

Note: This experiment has been set up to first generate the runs and then play them with different configuration (it has generated the expeRun dataset). To strictly repeat it, it is necessary to reuse the same input and not generate a new one. Hence, the input is saved in the repository, the generating line above must be commented (including in Docker).

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun.zip or Then unzip expeRun.zip -d input

Raw results¶

results/
  4-10000-add.txt
  4-10000-refine.txt
  4-10000-delete.txt
  4-10000-addjoin.txt
  4-10000-refine.tsv
  4-10000-delete.tsv
  4-10000-nothing.tsv
  4-10000-replace.txt
  4-10000-refadd.txt
  4-10000-add.tsv
  4-10000-replace.tsv
  4-10000-refadd.tsv
  4-10000-addjoin.tsv
  4-10000-nothing.txt

Analysis¶

Operator	Success rate	Size	Incoherence	Semantic precision	Semantic F-measure	Semantic recall	Convergence
nothing	0.26	54	0.34	0.11	0.20	0.89	9999
delete	1.00	6	0.00	1.00	0.13	0.07	154
replace	0.99	12	0.02	0.96	0.25	0.14	1224
refine	0.99	20	0.03	0.95	0.38	0.24	489
add	0.98	32	0.16	0.79	0.52	0.39	602
addjoin	0.99	32	0.16	0.79	0.52	0.39	1526
refadd	0.99	49	0.15	0.79	0.67	0.58	1554

Discussion¶

Key points¶

The results are commensurate with 20170214a-NOOR and 20170214b-NOOR
Except that now, as expected, final results for add and addjoin are the same
As a consequence results of refadd are even better (they used addjoin)
Applying the same set of events shows that addjoin converges faster, but add reaches the same resulting alignments

Further experiments¶

Redo all experiments with expansion, relaxation, generative and start empty.

Remarks:¶

The fact that the worse case convergence for addjoin is the same as for add may simply be related to the case of the last converging example out of 10. To be checked.

This file can be retrieved from URL https://sake.re/20180308-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180308-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180308-LOG.zip

See original markdown (20180308-NOOR.md) or HTML (20180308-NOOR.html) files.