Experiment 20170215a-NOOR

Relaxation (80%) increases precision and reach consistency [[euzenat2017a]]
Results for addjoin and refadd are underestimated (see [20180311-NOOR])

Experiment design

DockerOS DockerEXP

Date: 20170215

Hypotheses: Introducing the capability for agent to sometimes ignore the most specific applicable correspondences (with given immediate consumption probability) will improve precision by fighting shadowing.

Variation of: 20170209-NOOR

4 agents; 10 runs; 10000 games

Adaptation operators: delete replace refine add addjoin refadd

Experimental setting: Repetition of [20170208-NOOR] and [20170209-NOOR] with consuption probability set to 80%

Variables

controled variables: ['revisionModality']

dependent variables: ['srate', 'size', 'inc', 'prec', 'fmeas', 'rec', 'conv']

Experiment

Date: 20170215

Performer: Jérôme Euzenat (INRIA)

Lazy lavender hash: dc342dca26a5a6feff895b2a1829b04e7dfaa5ae

Classpath: lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:.

OS: stretch

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

for mod in ${OPS}
do
   java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=${NBAGENTS} -DnbIterations=${NBITERATIONS} -DnbRuns=${NBRUNS} -DreportPrecRec -DrevisionModality=${mod} -DimmediateRatio=80
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0 HotSpot

Raw results

results/
  20170215a-NOOR-4-10000-add-im80.txt
  20170215a-NOOR-4-10000-addjoin-im80.tsv
  20170215a-NOOR-4-10000-refine-im80.txt
  20170215a-NOOR-4-10000-add-im80.tsv
  20170215a-NOOR-4-10000-replace-im80.tsv
  20170215a-NOOR-4-10000-addjoin-im80.txt
  20170215a-NOOR-4-10000-refadd-im80.tsv
  20170215a-NOOR-4-10000-delete-im80.txt
  20170215a-NOOR-4-10000-replace-im80.txt
  20170215a-NOOR-4-10000-refine-im80.tsv
  20170215a-NOOR-4-10000-delete-im80.tsv
  20170215a-NOOR-4-10000-refadd-im80.txt

Analysis

With the standard modalities:

Operator Success rate Size Incoherence Semantic precision Semantic F-measure Semantic recall Convergence
delete 1.00 6 0.00 1.00 0.13 0.07 622
replace 0.99 12 0.00 1.00 0.21 0.12 947
refine 0.99 19 0.00 1.00 0.34 0.20 1708
add 0.98 25 0.00 1.00 0.37 0.23 2159
addjoin 0.99 19 0.00 1.00 0.33 0.20 563
refadd 0.98 40 0.00 1.00 0.61 0.44 1743

Compared 80% immediate consumption probability (20170215a-NOOR) to raw operators (20170214a-NOOR and 20170214b-NOOR):

  • In plain, raw operator semantic F-measure;
  • In dashed, 80% immediate consumption probability semantic F-measure.

(drawn from data scaled every 100 measures)

Discussion

Key points

  • The results converge in all cases to 100% precision and 0% incoherence.
  • The behaviour of addjoin as related to add is still unclear
  • Recall and F-measure are however lower

Further experiments

  • sortout the add/addjoin discrepancy
  • associate it with the expansion operator (see 20170215b-NOOR)

This file can be retrieved from URL https://sake.re/20170215a-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20170215a-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)

See original markdown (20170215a-NOOR.md) or HTML (20170215a-NOOR.html) files.