Experiment 20140208-NOOR

Agents achieve better syntactic F-measure than logical repairer (LogMap/Alcomo)

Experiment design

DockerOS DockerEXP

Date: 2014-02-08

Hypotheses: This is a revision of 20140207-NOOR with the use of Ontowrap which has revealed flaws in reference alignments and thus incorrect F-measures (and certainly revision). The network can achieve null incoherence degree. The network can revise better (in terms of F-measure) than LogMap and Alcomo.

Variation of: 20140207-NOOR

Experimental setting: Same as [20140207-NOOR] but it is possible to compare with LogMap and Alcomo.

Variables

controled variables: []

dependent variables: ['srate', 'fmeas', 'inc', 'conv']

Experiment

Date: 2014-02-08

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: 9cbfdf7e043a4dd82d8d4f7d06bd70ec6527dd0c

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: wheezy

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

java -Xms500M -Xmx1G -Dlog.level=DEBUG -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DreportIncoherence=1

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor. .

Execution environment: MacBook Pro 15, C2D 2.66GHz, 8GB - MacOS X 6.8 - Java 1.6.0 HotSpot

Raw results

results/
  20140208-NOOR.txt
  20140208-NOOR.tsv

Analysis

Initial results

Average degrees (Results from LogMap and Alcomo are reported from 20140208-NOOR.txt):

measure F-measure Incoherence degree
Initial 0.30 0.33
LogMap 0.32 0.21
Alcomo 0.30 0.00
Final 0.38 0.01

Invalid inhoherence for LogMap (due to a bug in the encoding of the network for computing incoherence): 2014-02-22

Discussion

After, 100+ steps the network has reduced incoherence to nearly null.

Key points:

F-measure passes very quickly above that of LogMap and Alcomo (simply because agents know exactly what to revise).

F-measure now goes from 30% to 40%. This is higher than the previous experiments (invalid).

The explanations given for 20140207-NOOR still hold

  • All actions only remove correspondences, but above all
  • F-measure is purely syntactic
  • Alignments are not symmetric, then revision can only apply on half of them

Incoherence degree is non null because:

  • Alignments are not symmetric

Further experiments:

We are in position to implement the other experiments:

  • Make the matrix symmetric or implements "alignment" at each stage
  • Use semantic precision and recall.

This file can be retrieved from URL https://sake.re/20140208-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20140208-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20140208-NOOR.md) or HTML (20140208-NOOR.html) files.