Experiment 20140311-NOOR

Agents with delete/replace/add achieve better syntactic F-measure than logical repairer [euzenat2014c]

Experiment design

DockerOS DockerEXP

Date: 2014-03-11

Hypotheses: Confirm the hypothesis that in 20140306-NOOR the difference between LogMap and Alcomo and the AlignmentAdjustingAgent approach cannot be found in simple terms of precision and recall, but are due to the semantic precision and recall.

Variation of: 20140306-NOOR

4 agents; 10 runs; 2000 games

Adaptation operators: delete replace add

Experimental setting: Very same setting as [20140306-NOOR] with classical precision and recall,but the lines implementing semantic precision and recall have been replaced by those with classical precision and recall (commented in the software)

Variables

controled variables: ['revisionModality']

dependent variables: ['srate', 'size', 'fmeas', 'conv']

Experiment

Date: 2014-03-11

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: 2aec5fe496c2b95760dba0ef87e82ac13264879b

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: wheezy

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

for op in ${OPS}
do
	java -Xms500M -Xmx1G -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=${op} -DnbRuns=${NBRUNS} -DnbAgents=${NBAGENTS} -DnbIterations=${NBITERATIONS} -o results/${LABEL]-4${op}.tsv > results/${LABEL]-4${op}.txt
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

results/
  20140311-NOOR-4replace.tsv
  20140311-NOOR-4add.txt
  20140311-NOOR-4replace.txt
  20140311-NOOR-4delete.tsv
  20140311-NOOR-4delete.txt
  20140311-NOOR-4add.tsv

Analysis

Initial results

Modality Size Success rate Incoherence Syntactic F-measure Convergence
Reference 70 nan 0.00 1.00 nan
Initial 54 nan 0.49 0.14 nan
delete 6 0.98 0.00 0.16 147
replace 6 0.95 0.00 0.16 373
add 12 0.89 0.00 0.16 997
Alcomo 27 nan 0.00 0.14 1
LogMap 36 nan 0.00 0.14 1

Discussion

Key points:

The hypothesis is confirmed:

  • Measures are far lower for all repair mechanisms;
  • With syntactic precision and recall, the add modalities (and in fact all the others) are better than the baseline algorithms.

This file can be retrieved from URL https://sake.re/20140311-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140311-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20140311-NOOR.md) or HTML (20140311-NOOR.html) files.