Experiment 20180531-NOOR¶

Starting with a realistic alignment size improves the final size (hence recall and F-measure)

Experiment design¶

Date: 20180723

Hypotheses: Starting with a realistic alignment size improves the final size (hence recall and F-measure)

Variation of: 20180307-NOOR

4 agents; 10 runs; 10000 games

Adaptation operators: delete replace refine add addjoin refadd

Experimental setting: Same as [[20180307-NOOR]] replaying the same runs as [[20180308-NOOR]] (with different initial alignments) and after fixing addjoin, expansion and real size estimation.

Variables¶

controled variables: ['revisionModality', 'expandAlignments', 'nonRedundancy', 'immediateRatio']

dependent variables: ['srate', 'size', 'inc', 'prec', 'fmeas', 'rec', 'conv']

Experiment¶

Date: 20180723

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: d50e70f87bca76951ec2f149dc8ae1d42b9a1a28

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

for op in ${OPS}
do
   bash scripts/runexp.sh -d 4-10000-${op}-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op}
   bash scripts/runexp.sh -d 4-10000-${op}-im80-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DimmediateRatio=80
   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy
   bash scripts/runexp.sh -d 4-10000-${op}-clever-nr-im80-real java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor ${OPT} ${LOADOPT} -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80

done

exit

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results¶

Analysis¶

Systematic comparison of realistic (plain) with plain (20180308-NOOR), clever-nr (20180529-NOOR), im80 (20180311-NOOR) and clever-nr-im80 (20180530-NOOR) (all dashed):

Relaxation	Expansion	Operator	Success rate	Size	Incoherence	Semantic precision	Semantic F-measure	Semantic recall	Convergence
0	0	nothing	0.70	92	0.35	0.25	0.39	0.89	1
0	0	delete	0.99	26	0.06	0.91	0.33	0.20	1030
0	0	replace	0.99	26	0.06	0.91	0.33	0.20	1030
0	0	refine	0.99	34	0.06	0.92	0.46	0.31	1030
0	0	add	0.99	49	0.19	0.78	0.59	0.48	1198
0	0	addjoin	0.99	49	0.19	0.78	0.59	0.48	958
0	0	refadd	0.99	63	0.19	0.78	0.70	0.64	958
0	1	delete	0.98	47	0.13	0.82	0.56	0.42	2039
0	1	replace	0.98	44	0.13	0.84	0.54	0.39	1078
0	1	refine	0.98	54	0.14	0.82	0.64	0.52	2681
0	1	add	0.98	79	0.25	0.71	0.73	0.75	2064
0	1	addjoin	0.99	79	0.24	0.72	0.74	0.75	1517
0	1	refadd	0.99	87	0.26	0.70	0.76	0.84	1154
1	0	delete	0.99	23	0.00	1.00	0.26	0.15	1659
1	0	replace	0.99	23	0.00	1.00	0.26	0.15	1648
1	0	refine	0.99	33	0.00	1.00	0.41	0.26	1659
1	0	add	0.99	40	0.00	1.00	0.46	0.30	5642
1	0	addjoin	0.99	40	0.00	1.00	0.46	0.30	3635
1	0	refadd	0.99	60	0.00	1.00	0.69	0.52	3256
1	1	delete	0.98	37	0.00	1.00	0.41	0.26	1104
1	1	replace	0.98	37	0.00	1.00	0.42	0.26	2078
1	1	refine	0.98	47	0.00	1.00	0.56	0.39	2887
1	1	add	0.97	66	0.00	1.00	0.68	0.52	8959
1	1	addjoin	0.98	65	0.00	1.00	0.68	0.51	8338
1	1	refadd	0.98	82	0.00	1.00	0.83	0.71	7733

Discussion¶

Analyst: Jérôme Euzenat (INRIA) (2018-08-16)

Key points¶

The realistic alignments obtained tend to have a higher size
They seem to converge faster
They indeed obtain better results:
- Precision reaches 100% when used with relaxation
- Recall and F-measure seems to be at least higher (2 points exception for refadd may be attributed to randomness)

All this is comparable to the results obtained in 20180307-NOOR.

Further experiments¶

Would be interesting to see what happens with the generative modality

This file can be retrieved from URL https://sake.re/20180531-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20180531-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20180531-NOOR.zip

See original markdown (20180531-NOOR.md) or HTML (20180531-NOOR.html) files.