20160315-NOOR

Bug in LogMap confirmed and hence LogMap results (see 20170227-NOOR)

Experiment design

Hypothesis: It is hypothesized that the repairing systems will reach increasingly better F-measures if they can benefit from the work made by cultural evolution

Experimental setting: Same as 20140304-NOOR with only 5 agents, 6000 iterations and a repeated repair every 400 games

Experiment

Experimenter: Jérôme Euzenat (INRIA)

Date: 2016-03-15

Lazy lavender hash: abaab185fe5cf9412ac3f90d8be23436500b7c06

Parameters: params.sh

Command line (script.sh):

 java -Xms500M -Xmx1G -cp lib/lazylav/ll.jar:lib/slf4j/logback-classic-1.0.9.jar:lib/slf4j/logback-core-1.0.9.jar:. -Dlog.level=INFO fr.inria.exmo.lazylavender.engine.Monitor -DrevisionModality=add -DnbRuns=10 -DnbAgents=5 -DnbIterations=6000 -DrepeatRepairs=400 > log5-6000.txt

Class used: NOOEnvironment, AlignmentAdjustingAgent, AlignmentRevisionExperiment, ActionLogger, AverageLogger, Monitor.

Execution environment: HP Folio 1040 G2, i7 2.6GHz, 8GB - Xubuntu 14.04 - Java 1.8.0_74 HotSpot 64-Bit

Raw results

20160315-NOOR-Log5-6000.tsv 20160315-NOOR-Log5-6000.txt 20160315-NOOR-output.txt

Result exploration

Alcomo LogMap Cultural evolution
game #corresp F-meas #corresp F-meas #corresp F-meas
0 73.1 0.1734772702 98.7 0.1599799054 170 0.1109946944
400 138.6 0.1392437083 130.6 0.1335113371 237.8 0.1223631356
800 118.1 0.1419336906 109.8 0.1361564735 198.4 0.1299611628
1200 98.1 0.1453711197 89.9 0.1434523097 160.3 0.1455042706
1600 80.3 0.1522552265 75.2 0.1481536428 122.8 0.1688372395
2000 64.1 0.1655167092 58.9 0.1644720434 94.4 0.2006516114
2400 52.7 0.1597021376 49.4 0.1643910656 76.1 0.2201501037
2800 45.1 0.1567526445 42.2 0.16584997 64.4 0.2336271293
3200 41 0.1466888939 38.2 0.1535234733 58.7 0.2341792917
3600 39.7 0.1439677631 36.5 0.1512732386 56.9 0.2364494257
4000 39.3 0.1435393832 36.6 0.1500871028 56.4 0.2371001664
4400 39.2 0.1455151159 36.3 0.1493567993 56 0.236920568
4800 38.9 0.1430536656 36 0.146736982 55.7 0.2366626944
5200 38.8 0.1426206581 35.9 0.1459479574 55.6 0.2367702742
5600 38.7 0.1426787531 35.8 0.1460227983 55.5 0.2370431633

All averaged over 10 runs

Analysis

Key points: Looking closely at the results, many things can be learnt:

This has been reproduced on the same basis on 20170208b-NOOR|2016-03-28.

There was a bug in LogMap and another in the way Alcomo evaluated incoherence.

Further experiments:


This file can be retrieved from URL https://sake.re/20160315-NOOR

It is possible to check out the repository by cloning https://felapton.inrialpes.fr/cakes/20160315-NOOR.git

This experiment has been transferred from its initial location at https://gforge.inria.fr.