20171218-NOOR

2018-05-16: Under scrutiny because expansion did not work since commit 562f5c4b96893d30e2dc9fd59d935e013ce5219f

Experiment design

Hypothesis:

Experimental setting: Same as 20171128-NOOR with 70000 and 100000 iterations only on:

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2017-12-18 to 2017-12-30

Lazy lavender hash: ab0edf530d7d832b2da9b6f208f729917fbe8e59

Parameters: params.sh

Command line (script.sh):

for it in 70000 100000
do 
   for op in delete replace refine add addjoin refadd
   do
      java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=${op} -DexpandAlignments=clever -DnonRedundancy -Dgenerative
   done
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -Dgenerative -Dstartempty
   java -Dlog.level=INFO -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.2.3.jar:lib/slf4j/logback-core-1.2.3.jar:. fr.inria.exmo.lazylavender.engine.Monitor -DnbAgents=5 -DnbIterations=${it} -DnbRuns=10 -DreportPrecRec -DrevisionModality=add -DexpandAlignments=clever -DnonRedundancy -DimmediateRatio=80 -Dgenerative -Dstartempty
  done
done

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor

Execution environment: Debian Linux virtual machine configured with four processors and 20GB of RAM running under a Dell PowerEdge T610 with 4*Intel Xeon Quad Core 1.9GHz E5-2420 processors, under Linux ProxMox 2 (Debian). - Java 1.8.0_131-b11 HotSpot

Raw results

5-100000-add-clever-nr-gen-empty.tsv 5-100000-add-clever-nr-gen-empty.txt 5-100000-add-clever-nr-gen.tsv 5-100000-add-clever-nr-gen.txt 5-100000-add-clever-nr-im80-gen-empty.tsv 5-100000-add-clever-nr-im80-gen-empty.txt 5-100000-add-clever-nr-im80-gen.tsv 5-100000-add-clever-nr-im80-gen.txt 5-100000-addjoin-clever-nr-gen.tsv 5-100000-addjoin-clever-nr-gen.txt 5-100000-delete-clever-nr-gen.tsv 5-100000-delete-clever-nr-gen.txt 5-100000-refadd-clever-nr-gen.tsv 5-100000-refadd-clever-nr-gen.txt 5-100000-refine-clever-nr-gen.tsv 5-100000-refine-clever-nr-gen.txt 5-100000-replace-clever-nr-gen.tsv 5-100000-replace-clever-nr-gen.txt 5-70000-add-clever-nr-gen-empty.tsv 5-70000-add-clever-nr-gen-empty.txt 5-70000-add-clever-nr-gen.tsv 5-70000-add-clever-nr-gen.txt 5-70000-add-clever-nr-im80-gen-empty.tsv 5-70000-add-clever-nr-im80-gen-empty.txt 5-70000-add-clever-nr-im80-gen.tsv 5-70000-add-clever-nr-im80-gen.txt 5-70000-addjoin-clever-nr-gen.tsv 5-70000-addjoin-clever-nr-gen.txt 5-70000-delete-clever-nr-gen.tsv 5-70000-delete-clever-nr-gen.txt 5-70000-refadd-clever-nr-gen.tsv 5-70000-refadd-clever-nr-gen.txt 5-70000-refine-clever-nr-gen.tsv 5-70000-refine-clever-nr-gen.txt 5-70000-replace-clever-nr-gen.tsv 5-70000-replace-clever-nr-gen.txt

Result exploration

Table for expansion + generation:

testiterationssuccess
rate
network
size
incoherence
degree
semantic
precision
semantic
F-measure
semantic
recall
maximum
convergence
delete-clever-nr-gen200000.892340.001.000.500.3319903
450000.952310.001.000.510.3442425
700000.972340.001.000.510.3441883
1000000.982330.001.000.510.3464487
replace-clever-nr-gen200000.892300.000.990.520.3519962
450000.952290.001.000.520.3543617
700000.972260.001.000.520.3547963
1000000.982270.001.000.490.3345471
refine-clever-nr-gen200000.892310.010.990.560.3919958
450000.952280.000.990.580.4143810
700000.972290.010.990.580.4151666
1000000.982290.010.990.590.4248918
add-clever-nr-gen200000.832590.100.760.670.6019998
450000.902310.100.860.700.6044999
700000.932250.090.900.710.5869817
1000000.952230.100.890.710.5998479
addjoin-clever-nr-gen200000.902290.030.970.630.4619987
450000.952280.030.970.620.4644995
700000.972300.030.970.640.4869659
1000000.982280.030.970.630.4661415
refadd-clever-nr-gen200000.912340.060.930.700.5619998
450000.952340.060.930.700.5643466
700000.972350.050.940.690.5455635
1000000.982310.050.930.690.5549768

Table for add + expansion + generation + other modalities:

testiterationssuccess
rate
network
size
incoherence
degree
semantic
precision
semantic
F-measure
semantic
recall
maximum
convergence
add-clever-nr-gen200000.832590.100.760.670.6019998
450000.902310.100.860.700.6044999
700000.932250.090.900.710.5869817
1000000.952230.100.890.710.5998479
add-clever-nr-im80-gen200000.842540.020.880.580.4320000
450000.902350.010.960.600.4344999
700000.932250.000.990.600.4369933
1000000.952200.001.000.590.4299900
add-clever-nr-gen-empty200000.882790.000.930.520.3619989
450000.932610.001.000.530.3644930
700000.962600.001.000.520.3663670
1000000.972600.001.000.540.3762145
add-clever-nr-im80-gen-empty200000.882770.000.940.530.3719999
450000.932610.001.000.530.3644999
700000.962600.001.000.520.3661292
1000000.972600.001.000.540.3768328

Below are the evolution curves: first for all operators with expansion + generation; then for add with expansion + generation + other modalities.

(drawn from data scaled every 100 measures)

Observations:

Analysis

Key points:

Further experiments:

Remarks:


This file can be retrieved from URL https://sake.re/20171218-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20171218-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr.

The original, unaltered associated zip file can be obtained from https://files.inria.fr/sakere/gforge/20171218-NOOR.zip