Experiment 20140407-NOOR

Increasing initial number of correspondences has little effect [euzenat2014c]

Experiment design

DockerOS DockerEXP

Date: 2014-04-07

Hypotheses: Increasing the number of initial correspondences improves the performance of agents (because it leaves them more chances to find correct correspondences)

Variation of: 20140305-NOOR

4 agents; 10 runs; 2000 games

Adaptation operators: add

Experimental setting: Same as [20140305-NOOR] with: Number of initial correspondences set to 6, 8, 10, 11, 12

Variables

controled variables: ['nbCorresp']

dependent variables: ['srate', 'size', 'inc', 'fmeas', 'conv']

Experiment

Date: 2014-04-07

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: 89f0656ffc28a6c4560a7a55cf1508d866528a96

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: wheezy

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

for nc in ${NBCORRESP}
do
	java -Xms500M -Xmx1G -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=${OPS} -DnbRuns=${NBRUNS} -DnbAgents=${NBAGENTS} -DnbIterations=${NBITERATIONS} -DnbCorresp=${nc} -o results/${LABEL}-${OPS}${NBAGENTS}-${NBITERATIONS}-${nc}.tsv > results/${LABEL}-${OPS}${NBAGENTS}-${NBITERATIONS}-${nc}.txt
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

results/
  20140407-NOOR-add4-2000-12.txt
  20140407-NOOR-add4-2000-6.tsv
  20140407-NOOR-add4-2000-11.txt
  20140407-NOOR-add4-2000-10.txt
  20140407-NOOR-add4-2000-12.tsv
  20140407-NOOR-add4-2000-8.txt
  20140407-NOOR-add4-2000-10.tsv
  20140407-NOOR-add4-2000-11.tsv
  20140407-NOOR-add4-2000-6.txt
  20140407-NOOR-add4-2000-8.tsv

Analysis

Initial results

Measure Size Success rate Incoherence Semantic F-measure Size Convergence Semantic F-measure Size Semantic F-measure Size
#Corresp Reference Final Final Final Final Final LogMap LogMap Alcomo Alcomo
6 42.00 0.92 0.00 0.20 10.80 1497 0.26 31.00 0.27 24.20
8 54.00 0.89 0.00 0.23 16.10 1392 0.25 34.30 0.25 28.70
10 66.00 0.86 0.00 0.23 16.90 1969 0.24 42.70 0.24 32.30
11 72.00 0.84 0.00 0.23 16.90 1677 0.23 42.70 0.24 33.70
12 78.00 0.83 0.00 0.25 21.10 1764 0.25 47.10 0.23 35.30

The size of the reference alignment is 70 (globally, which sets it between 10 and 12 initial correspondences. After 12, it is not possible to generate more correspondences without being redundant (this is forbidden by the program). The test performed before (eg 20140305-NOOR), are with 54 correspondences, hence nbCorresp=8.

Discussion

Key points

  • The increase of the number of correspondences augments the chances to retain a correct correspondences and thus increases the quality of alignments provided by agents.
  • This is also measured with respect to LogMap and Alcomo which have, on their side, more chances to retain an incorrect correspondence. So their F-measure decreases while that of the agents increases.

This file can be retrieved from URL https://sake.re/20140407-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140407-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20140407-NOOR.md) or HTML (20140407-NOOR.html) files.