Experiment 20140225-NOOR

add > replace > delete

Experiment design

DockerOS DockerEXP

Date: 2014-02-25

Hypotheses: - The add modality reaches a better F-measure than replace which reaches a better F-measure than delete. - The add modality on 4 agents outperforms LogMap and Alcomo

Variation of: 20140224-NOOR

4 agents; 10 runs; 500 games

Adaptation operators: delete replace add

Experimental setting: This is the same setting as [20140224-NOOR], except that the exercise run 10 times for 500 iteration

Variables

controled variables: ['revisionModality']

dependent variables: ['srate', 'fmeas', 'conv']

Experiment

Date: 2014-02-25

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: 402eae910f33cf73a4663fe83308d0b0be6d7b5b

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: wheezy

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

for op in ${OPS}
do
java -Xms500M -Xmx1G -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=${op} -DnbRuns=10 -DnbAgents=4 -DnbIterations=500 -o ~/plot12${op}.table 
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

results/
  20140225-NOOR.txt

Analysis

Initial results

Result files are not available. Hence the data below are those of the original notebook.

  • Plain lines are success rates; dashed lines are F-measure
  • In blue, add;
  • In green, replace;
  • In red, delete;
  • In black, the Alcomo baseline.
modality success convergence Incoherence degree F-measure
Initial 0.49 0.20
Delete 0.90 445 0.0 0.16
Replace 0.81 489 0.0 0.18
Add 0.62 500+ 0.0 0.26
LogMap 1 0.0 0.25
Alcomo 1 0.0 0.26

Discussion

Key points:

  • The situation is better than with three agents in which case very often the revision deletes all the correspondences which are not top=top
  • Clearly add outperforms replace which is slightly better than delete
  • Add has not converged after 500 iterations (on 10 runs), it takes around 1200 iterations
  • Again, the baseline is reached, but not improved.

Among the runs, there is a lot of variety, any of the system may reach the highest F-measure (with the add modality).

Further experiments:

  • Further increase the number of agents
  • Try with more elaborate modalities
This file can be retrieved from URL [https://sake.re/20140225-NOOR](https://sake.re/20140225-NOOR)

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140225-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20140225-NOOR.md) or HTML (20140225-NOOR.html) files.