Experiment 20140304-NOOR

Agents improves semantic F-measure over logical repairer there are more agents/features [euzenat2014b]

Experiment design

DockerOS DockerEXP

Date: 2014-03-04

Hypotheses: If one leaves the process to last long enough, it will provide a better F-measure than logical repair.

Variation of: 20140225-NOOR

3 4 5 6 agents; 10 runs; 10000 games

Adaptation operators: add

Experimental setting: As [20140225-NOOR], except that: The number of iterations is 10000; The number of agents goes from 3 to 6; The software now logs the total number of correspondences.

Variables

controled variables: ['nbAgents']

dependent variables: ['srate', 'size', 'inc', 'fmeas', 'conv']

Experiment

Date: 2014-03-04

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: 2aec5fe496c2b95760dba0ef87e82ac13264879b

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: wheezy

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

for NA in ${NBAGENTS}
do
    java -Xms500M -Xmx1G -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=${NBRUNS} -DnbAgents=${NA} -DnbIterations=${NBITERATIONS} > log${NA}-${NBITERATIONS}.txt 
done

exit

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Takes a whole night.

Raw results

results/
  20140304-NOOR-Log6-10000.tsv
  20140304-NOOR-Log4-10000.txt
  20140304-NOOR-Log3-10000.txt
  20140304-NOOR-Log4-10000.tsv
  20140304-NOOR-Log6-10000.txt
  20140304-NOOR-Log5-10000.tsv
  20140304-NOOR-Log5-10000.txt
  20140304-NOOR-Log3-10000.tsv

Analysis

Initial results

Measure Size Incoherence Semantic F-measure Convergence Success rate
NbAgents Reference Initial LogMap Alcomo Final Initial LogMap Alcomo Final Initial LogMap Alcomo Final Final Final
3 15.00 15.00 12.00 10.30 3.00 0.31 0.00 0.00 0.00 0.32 0.35 0.36 0.33 254 1.00
4 70.00 54.00 35.70 28.70 12.60 0.47 0.00 0.00 0.00 0.20 0.26 0.26 0.22 1538 0.98
5 250.00 170.00 94.70 71.70 47.40 0.58 0.00 0.00 0.00 0.11 0.18 0.17 0.24 5163 0.90
6 783.00 495.00 234.30 182.10 224.20 0.63 0.00 0.00 0.00 0.06 0.12 0.11 0.14 9936 0.67

Discussion

Key points:

  • The behaviour is long and erratic
  • In the long run, it seems that indeed average F-measure is better.

Complement:

On 2014-03-11, a supplementary 10 runs with 5 agents have been performed for confirmation purposes. It provided the very same results.

This file can be retrieved from URL [https://sake.re/20140304-NOOR](https://sake.re/20140304-NOOR)

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140304-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20140304-NOOR.md) or HTML (20140304-NOOR.html) files.