Experiment 20180601-NOOR

The generative modality increases results quality (recall) with respect to expansion
Repeated in [20190624-NOOR]

Experiment design

DockerOS DockerEXP

Date: 2018-05-30

Hypotheses: Generating correspondences when needed should increase results quality (recall) with respect to expansion

Variation of: 20170530-NOOR

4 agents; 10 runs; 10000 games

Adaptation operators: delete replace refine add addjoin refadd

Experimental setting: Same as [[20170530-NOOR]] replaying the same runs as [[20180308-NOOR]] and after fixing addjoin and expansion.

Variables

controled variables: ['revisionModality', 'immediateRatio']

dependent variables: ['srate', 'size', 'inc', 'prec', 'fmeas', 'rec', 'conv']

Experiment

Date: 2018-06-01

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: d50e70f87bca76951ec2f149dc8ae1d42b9a1a28

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

for op in ${OPS}
do
   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-gen java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative
   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-im80-gen java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative
#   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-gen-empty java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative -DstartEmpty
#   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-im80-gen-empty java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative -DstartEmpty

done

exit

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun.zip or DOI Then unzip expeRun.zip -d input

Raw results

Analysis

Systematic comparison of clever-nr (dashed, 20180529-NOOR) to clever-nr-gen (plain) and clever-nr-im80 (dashed, 20180530-NOOR) to clever-nr-im80-gen (plain):

Relaxation Operator Success rate Size Incoherence Semantic precision Semantic F-measure Semantic recall Convergence
0 delete 0.94 59 0.12 0.85 0.67 0.55 9521
0 replace 0.94 61 0.13 0.84 0.71 0.62 9628
0 refine 0.94 71 0.16 0.80 0.76 0.71 7761
0 add 0.91 75 0.23 0.73 0.77 0.83 9068
0 addjoin 0.95 77 0.23 0.72 0.76 0.81 6319
0 refadd 0.94 78 0.23 0.70 0.76 0.85 7848
1 delete 0.93 51 0.00 1.00 0.60 0.43 9875
1 replace 0.94 55 0.00 1.00 0.64 0.47 9731
1 refine 0.93 65 0.00 1.00 0.76 0.62 9977
1 add 0.91 67 0.00 1.00 0.79 0.65 9878
1 addjoin 0.95 69 0.00 1.00 0.80 0.67 9730
1 refadd 0.94 76 0.00 1.00 0.86 0.76 7860

Discussion

Analyst: Jérôme Euzenat (INRIA) (2018-08-16)

Key points

  • The generative modality always generates larger alignments
  • It is always slower to converge
  • Always generates a higher F-measure and recall
  • Has a lower precision wrt clever-nr, but preserves the same 100% precision as -im80

These are the same conclusions as 20170530-NOOR

Further experiments

none


This file can be retrieved from URL https://sake.re/20180601-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180601-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180601-NOOR.zip

See original markdown (20180601-NOOR.md) or HTML (20180601-NOOR.html) files.