Experiment 20170209-NOOR

refadd > addjoin > refine
Results for addjoin and refadd are underestimated (see [20180308-NOOR])

Experiment design

DockerOS DockerEXP

Date: 20170209

Hypotheses: H1: Addjoin is faster to converge than add and achieves the same results, H2: refine is better than replace (in [20170208-NOOR] for instance), H3: refadd is better than add and refine (especially with better recall)

Variation of: 20170208-NOOR

4 agents; 10 runs; 10000 games

Adaptation operators: refine addjoin refadd

Experimental setting: Exactly the same as [20170208-NOOR] with new operators

Variables

controled variables: ['revisionModality']

dependent variables: ['srate', 'size', 'inc', 'prec', 'fmeas', 'rec', 'conv']

Experiment

Date: 20170209

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: 738f55c3e7dc6fc16ebcfe070adc314dd4703d21

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

for op in ${OPS}
do
	java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=${op} -DnbAgents=${NBAGENTS} -DnbIterations=${NBITERATIONS} -DnbRuns=${NBRUNS} -DreportPrecRec
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results

results/
  20170209-NOOR-4-10000-refadd.txt
  20170209-NOOR-4-10000-refine.txt
  20170209-NOOR-4-10000-refine.tsv
  20170209-NOOR-4-10000-addjoin.tsv
  20170209-NOOR-4-10000-refadd.tsv
  20170209-NOOR-4-10000-addjoin.txt

Analysis

Operator Success rate Size Incoherence Semantic precision Semantic F-measure Semantic recall Convergence
refine 0.99 20 0.03 0.95 0.37 0.23 727
addjoin 0.99 24 0.09 0.86 0.43 0.29 1523
refadd 0.99 40 0.09 0.86 0.61 0.47 950

For dealing with hypothesis H1, we compare with the result of add on 20170208-NOOR:

For dealing with hypothesis H2, we compare with the result of replace on 20170208-NOOR:

Discussion

Key points

  • Concerning H1, it seems not possible to conclude: addjoin seems indeed to converge faster on average while on the table, add has its maximum convergence step at 1235 inseatd of 1523 fo addjoin. Moreover, they achieve significant F-measure difference, so it would be better to compare them on the exact same scenarii.
  • Concerning H2, refine seems slower and achieve an higher F-measure than replace
  • With respect to H3, refadd is better in F-measure than refine and addjoin although it is slower to converge. It also, as expected, has better recall.

Further experiments

  • Test H1 with the exact same scenario.

This file can be retrieved from URL https://sake.re/20170209-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20170209-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20170209-NOOR.md) or HTML (20170209-NOOR.html) files.