Experiment 20180601-NOOR¶

The generative modality increases results quality (recall) with respect to expansion
Repeated in [20190624-NOOR]

Experiment design¶

Date: 2018-05-30

Hypotheses: Generating correspondences when needed should increase results quality (recall) with respect to expansion

Variation of: 20170530-NOOR

4 agents; 10 runs; 10000 games

Adaptation operators: delete replace refine add addjoin refadd

Experimental setting: Same as [[20170530-NOOR]] replaying the same runs as [[20180308-NOOR]] and after fixing addjoin and expansion.

Variables¶

controled variables: ['revisionModality', 'immediateRatio']

dependent variables: ['srate', 'size', 'inc', 'prec', 'fmeas', 'rec', 'conv']

Experiment¶

Date: 2018-06-01

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: d50e70f87bca76951ec2f149dc8ae1d42b9a1a28

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

for op in ${OPS}
do
   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-gen java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative
   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-im80-gen java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative
#   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-gen-empty java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative -DstartEmpty
#   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-im80-gen-empty java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative -DstartEmpty

done

exit

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Input: Input required for reproducibility can be retrieved from: https://files.inria.fr/sakere/input/expeRun.zip or Then unzip expeRun.zip -d input

Raw results¶

Analysis¶

Systematic comparison of clever-nr (dashed, 20180529-NOOR) to clever-nr-gen (plain) and clever-nr-im80 (dashed, 20180530-NOOR) to clever-nr-im80-gen (plain):

Relaxation	Operator	Success rate	Size	Incoherence	Semantic precision	Semantic F-measure	Semantic recall	Convergence
0	delete	0.94	59	0.12	0.85	0.67	0.55	9521
0	replace	0.94	61	0.13	0.84	0.71	0.62	9628
0	refine	0.94	71	0.16	0.80	0.76	0.71	7761
0	add	0.91	75	0.23	0.73	0.77	0.83	9068
0	addjoin	0.95	77	0.23	0.72	0.76	0.81	6319
0	refadd	0.94	78	0.23	0.70	0.76	0.85	7848
1	delete	0.93	51	0.00	1.00	0.60	0.43	9875
1	replace	0.94	55	0.00	1.00	0.64	0.47	9731
1	refine	0.93	65	0.00	1.00	0.76	0.62	9977
1	add	0.91	67	0.00	1.00	0.79	0.65	9878
1	addjoin	0.95	69	0.00	1.00	0.80	0.67	9730
1	refadd	0.94	76	0.00	1.00	0.86	0.76	7860

Discussion¶

Analyst: Jérôme Euzenat (INRIA) (2018-08-16)

Key points¶

The generative modality always generates larger alignments
It is always slower to converge
Always generates a higher F-measure and recall
Has a lower precision wrt clever-nr, but preserves the same 100% precision as -im80

These are the same conclusions as 20170530-NOOR

Further experiments¶

none

This file can be retrieved from URL https://sake.re/20180601-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180601-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180601-NOOR.zip

See original markdown (20180601-NOOR.md) or HTML (20180601-NOOR.html) files.