20160315-NOOR

Bug in LogMap confirmed and hence LogMap results (see 20170227-NOOR)

Experiment design

Hypothesis: It is hypothesized that the repairing systems will reach increasingly better F-measures if they can benefit from the work made by cultural evolution

Experimental setting: Same as 20140304-NOOR with only 5 agents, 6000 iterations and a repeated repair every 400 games

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2016-03-15

Lazy lavender hash: abaab185fe5cf9412ac3f90d8be23436500b7c06

Parameters: params.sh

Command line (script.sh):

 java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. -Dlog.level=INFO fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=5 -DnbIterations=6000 -DrepeatRepairs=400 > log5-6000.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: HP Folio 1040 G2, i7 2.6GHz, 8GB - Xubuntu 14.04 - Java 1.8.0_74 HotSpot 64-Bit

Raw results

20160315-NOOR-Log5-6000.tsv 20160315-NOOR-Log5-6000.txt 20160315-NOOR-output.txt

Result exploration

game	#corresp	F-meas	#corresp	F-meas	#corresp	F-meas
	Alcomo		LogMap		Cultural evolution
0	73.1	0.1734772702	98.7	0.1599799054	170	0.1109946944
400	138.6	0.1392437083	130.6	0.1335113371	237.8	0.1223631356
800	118.1	0.1419336906	109.8	0.1361564735	198.4	0.1299611628
1200	98.1	0.1453711197	89.9	0.1434523097	160.3	0.1455042706
1600	80.3	0.1522552265	75.2	0.1481536428	122.8	0.1688372395
2000	64.1	0.1655167092	58.9	0.1644720434	94.4	0.2006516114
2400	52.7	0.1597021376	49.4	0.1643910656	76.1	0.2201501037
2800	45.1	0.1567526445	42.2	0.16584997	64.4	0.2336271293
3200	41	0.1466888939	38.2	0.1535234733	58.7	0.2341792917
3600	39.7	0.1439677631	36.5	0.1512732386	56.9	0.2364494257
4000	39.3	0.1435393832	36.6	0.1500871028	56.4	0.2371001664
4400	39.2	0.1455151159	36.3	0.1493567993	56	0.236920568
4800	38.9	0.1430536656	36	0.146736982	55.7	0.2366626944
5200	38.8	0.1426206581	35.9	0.1459479574	55.6	0.2367702742
5600	38.7	0.1426787531	35.8	0.1460227983	55.5	0.2370431633

In dotted blue the average success rate;
In blue, the average semantic F-measure.
In green and red the F-measure obtained by Alcomo and LogMap by being launched against the current network every 400 games;

All averaged over 10 runs

Analysis

Key points: Looking closely at the results, many things can be learnt:

Very quickly the network becomes coherent. Hence, There is no added value of Alignment repair
The alignment repair algorithms are always destroying more correspondences than cultural evolution. This makes them never very useful.
However, it seems problematic that they still suppress correspondences in the end while the alignments should be consistent! There should be a bug!
Because the add modality has been used used alignments are multiplied at the beginning, but very soon, they are reduced. This explain the non monotonic behaviour of data.

This has been reproduced on the same basis on 20170208b-NOOR|2016-03-28.

There was a bug in LogMap and another in the way Alcomo evaluated incoherence.

Further experiments:

This file can be retrieved from URL https://sake.re/20160315-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20160315-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr (not available any more)