Experiment 20140305-NOOR

Software bugs in Alcomo, LogMap and reference alignments: incoherence and semantic revision figures are incorrect; conclusion hold (see 20170208-NOOR)

add > replace > delete [euzenat2014b]
Software bugs in Alcomo, LogMap and reference alignments: incoherence and semantic revision figures are incorrect; conclusion hold (see [20170208-NOOR])

Experiment design

DockerOS DockerEXP

Date: 2014-03-05

Hypotheses: del < repl < add.

Variation of: 20140225-NOOR

4 agents; 10 runs; 2000 games

Adaptation operators: delete replace add

Experimental setting: A remake of [20140225-NOOR], except that: The number of iterations is 2000.

Variables

controled variables: ['revisionModality']

dependent variables: ['srate', 'size', 'fmeas', 'conv']

Experiment

Date: 2014-03-05

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: 2aec5fe496c2b95760dba0ef87e82ac13264879b

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: wheezy

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

for op in ${OPS}
do
java -Xms500M -Xmx1G -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=${op} -DnbRuns=${NBRUNS} -DnbAgents=${NBAGENTS} -DnbIterations=${NBITERATIONS} > ${LABEL}-Log${NBAGENTS}-${NBITERATIONS}-${op}.txt
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

results/
  20140305-NOOR-Log4-2000-replace.txt
  20140305-NOOR-Log4-2000-delete.tsv
  20140305-NOOR-Log4-2000-replace.tsv
  20140305-NOOR-Log4-2000-add.txt
  20140305-NOOR-Log4-2000-add.tsv
  20140305-NOOR-Log4-2000-delete.txt

Analysis

Initial results

Modality Size Success rate Incoherence Semantic F-measure Convergence
Reference 70 nan 0.00 1.00 nan
Initial 54 nan 0.49 0.20 nan
delete 6 0.98 0.00 0.16 393
replace 6 0.95 0.00 0.16 644
add 13 0.89 0.00 0.23 1130
Alcomo 28 nan 0.00 0.26 1
LogMap 36 nan 0.00 0.26 1

Discussion

Key points:

As expected, del < repl < add.

This file can be retrieved from URL [https://sake.re/20140305-NOOR](https://sake.re/20140305-NOOR)

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140305-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20140305-NOOR.md) or HTML (20140305-NOOR.html) files.