Experiment 20210601-DOTG

Experiment design

Knowledge is transmitted between agents and across generations

Date: 20210601 (Yasser Bourahla)

10 runs; 200000 games

Setting : Agents start with random ontologies; agents adapt their knowledge to interaction failures with other agents (intra-generation knowledge transmission); agents provide training samples to their children to learn ontologies and interact with them (inter-generation knowledge transmission).

Hypothesis: Adding inter-generation transmission leads to higher accuracy than intra-generation transmission alone.

Variables

independent variables: ['period', 'transmission percentage', 'ontology transfer']

dependent variables: ['success rate by generation', 'accuracy']

Experiment

Date: 20210601 (Yasser Bourahla)

LazyLavender hash: 82ea8d4fc316c5197f1641205063d3a6d3a790a8

Link to lazylavender

Parameter file: (params.sh)

Executed command (script.sh):

#!/bin/bash

. params.sh

CURRDIR=$(pwd)
OUTPUT=${CURRDIR}/${DIRPREF}
cd lazylav

bash scripts/runexp.sh -p ${CURRDIR} -d ${DIRPREF} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.ExperimentalPlan -Dexperiment=fr.inria.exmo.lazylavender.decisiontaking.generations.GenerationExperiment ${OPT}  -DresultDir=${OUTPUT}

Experimental plan

The independent variables have been varied as follows:

period: [5001, 10001, 20001, 200001]
transmission percentage: [0.2, 0.4, 0.6, 0.8, 1.0]
ontology transfer: ['learn', 'random']

Raw results

Full results are available on zenodo

DOI DOI 10.5281/zenodo.5931033 10.5281/zenodo.5931033

Hypothesis testing

Adding inter-generation transmission leads to higher accuracy than intra-generation transmission alone.

influence of period
PR(>F) Significance
accuracy 0.000000 True 










***********************************************************
ANOVA results for accuracy
***********************************************************
df sum_sq mean_sq F PR(>F)
C(Q("period")) 3.0 3.921427 1.307142 186.613894 2.057993e-75
Residual 396.0 2.773794 0.007005 NaN NaN 
--------------------------------------
post-hoc (Tukey) test for period on accuracy
--------------------------------------
group1 group2 meandiff p-adj lower upper reject
0 10001 200001 -0.0524 0.0010 -0.0830 -0.0219 True
1 10001 20001 0.0315 0.0406 0.0009 0.0620 True
2 10001 5001 -0.2249 0.0010 -0.2555 -0.1944 True
3 200001 20001 0.0839 0.0010 0.0534 0.1144 True
4 200001 5001 -0.1725 0.0010 -0.2030 -0.1419 True
5 20001 5001 -0.2564 0.0010 -0.2869 -0.2259 True

Hypothesis validated for period =10001

Hypothesis validated for period =20001

Data exploration

Averaged results for each independent variable

period transmission percentage ontology transfer
5001 10001 20001 200001 0.2 0.4 0.6 0.8 1.0 learn random
genSrate 1 0.21 0.15 0.21 0.14 0.14 0.16 0.17 0.26 0.15 0.18 0.17
2000000 0.64 0.79 0.90 0.97 0.78 0.80 0.82 0.85 0.88 0.88 0.77
accuracy 1 0.17 0.17 0.17 0.17 0.17 0.17 0.17 0.17 0.17 0.17 0.17
2000000 0.67 0.90 0.93 0.84 0.83 0.83 0.84 0.84 0.83 0.83 0.84

Averaged results for each period/transmission percentage

period
5001 10001 20001 200001
ontology random 0.63 0.91 0.95 0.86
transmis 0.2 0.63 0.91 0.91 0.86
0.4 0.66 0.91 0.89 0.84
0.6 0.72 0.86 0.94 0.81
0.8 0.73 0.90 0.91 0.83
1.0 0.82 0.82 0.86 0.81

Transmission percentage effect

For period 5001
influence of transmission percentage
PR(>F) Significance
genSrate 0.000000 True
accuracy 0.000466 True



For period 10001
influence of transmission percentage
PR(>F) Significance
genSrate 0.000000 True
accuracy 0.137358 False



For period 20001
influence of transmission percentage
PR(>F) Significance
genSrate 0.000000 True
accuracy 0.130010 False



Inter-generation transmission by teaching vs by interaction alone

For period 5001
influence of ontology transfer
PR(>F) Significance
genSrate 0.000000 True
accuracy 0.000059 True



For period 10001
influence of ontology transfer
PR(>F) Significance
genSrate 0.000000 True
accuracy 0.046382 False



For period 20001
influence of ontology transfer
PR(>F) Significance
genSrate 0.000000 True
accuracy 0.000084 True