Experiment 20180311-NOOR

reproduction of results of 20170215a-NOOR: Relaxation (80%) increases precision and reach consistency; clearer behaviour of add/addjoin after bug correction (3fcf33/2018-03-05) affecting addjoin/refadd

Experiment design

DockerOS DockerEXP

Date: 2018-03-11

Hypothesis: Relaxation (80%) increases precision and reach consistency

Variation of: 20170215a-NOOR

4 agents; 10 runs; 10000 games

Adaptation operators: delete replace refine add addjoin refadd

Experimental setting: Same as the initial experiments [20170215a-NOOR] with fixed errors in the implementation of addjoin (affects addjoin and refadd); comparison on the exact same set of games (replayGames)

Variables

controled variables: ['revisionModality']

dependent variables: ['srate', 'size', 'inc', 'prec', 'fmeas', 'rec', 'conv']

Experiment

Date: 2018-03-11

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: b86d0cb89108ca78d0ead507098e14683e4d39fb

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash
# Script to run experiments in bactch on a remore machine
# Procedure:
# - zip the current directory: cd ..; zip -rq lazylav.zip lazylav
# - transfer it on the execution machine: scp lazylav.zip seals-4.inrialpes.fr:/scratch/LazyLav/
# - unzip on this machine: unzip lazylav.zip; cd lazylav
# - run it there: nohup bash myscript.sh < /dev/null 2> /dev/null > /dev/null &
# - results are found the directories passed as -d

. params.sh

for mod in ${OPS}
do
   bash scripts/runexp.sh -d results java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${mod} -o ${NBAGENTS}-${NBITERATIONS}-${mod}-im80.tsv
done

exit

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_151-b12 HotSpot

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun.zip or DOI Then unzip expeRun.zip -d input

Raw results

results/
  4-10000-refadd-im80.tsv
  4-10000-delete-im80.txt
  4-10000-replace-im80.txt
  4-10000-add-im80.tsv
  4-10000-refine-im80.txt
  4-10000-replace-im80.tsv
  4-10000-addjoin-im80.txt
  4-10000-add-im80.txt
  4-10000-addjoin-im80.tsv
  4-10000-refine-im80.tsv
  4-10000-refadd-im80.txt
  4-10000-delete-im80.tsv

Analysis

Operator Success rate Size Incoherence Semantic precision Semantic F-measure Semantic recall Convergence
delete 1.00 6 0.00 1.00 0.13 0.07 240
replace 0.99 12 0.00 1.00 0.22 0.13 1224
refine 0.99 19 0.00 1.00 0.35 0.21 360
add 0.98 26 0.00 1.00 0.41 0.26 2054
addjoin 0.99 26 0.00 1.00 0.41 0.26 2029
refadd 0.99 47 0.00 1.00 0.67 0.50 3376

Observations:

  • Note that the set of game events are the same as those of 20180308-NOOR

Discussion

Key points

  • The results are commensurate with 20170215a-NOOR
  • Except that now, as expected, final results for add and addjoin are the same
  • As a consequence results of refadd are even better (they use addjoin)
  • Applying the same set of events shows that addjoin converges faster, but add reaches the same resulting alignments
  • The results have the same relation with 20180308-NOOR (smaller size, 100% precision, lower recall) as 20170215a-NOOR has with 20170214a-NOOR and 20170214b-NOOR

Further experiments

  • Redo all experiments with expansion, generative and start empty.

Remarks

  • Now, again the convergence of add and addjoin are different (addjoin converging faster as expected)

This file can be retrieved from URL https://sake.re/20180311-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180311-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180311-LOG.zip

See original markdown (20180311-NOOR.md) or HTML (20180311-NOOR.html) files.