Experiment 20220516-DOLA

Experiment design

Agent adapt ontologies to agree on decision taking

Date: 20220516 (Yasser Bourahla)

5 runs; 100000 games

Setting : Agents learn decision trees (transformed into ontologies); get income from environment; adapt by splitting their leaf nodes

Hypothesis: Success rate converges to 1. Improve the average accuracy at the end of the experiment. Agents do not necessarily converge to the same ontology.

Variation of: 20200623-DOLA

Variables

independent variables: ['Environment bias coefficient', 'Inverse social bias coefficient', 'Frequency of adaptation']

dependent variables: ['success rate', 'accuracy', "avg agents with winner's decision", 'avg agents with correct decision', 'winner decision is correct']

Experiment

Date: 20220516 (Yasser Bourahla)

LazyLavender hash: 4b837b2619086143a5bbb798ac0b80b110f80342

Link to lazylavender

Parameter file: (params.sh)

Executed command (script.sh):

#!/bin/bash

. params.sh

CURRDIR=$(pwd)
OUTPUT=${CURRDIR}/${DIRPREF}
# cd ${LLPATH}
cd lazylav
# this sample runs ExperimentalPlan. It can be replaced with Monitor if parameters are not varied.
bash scripts/runexp.sh -p ${CURRDIR} -d ${DIRPREF} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.ExperimentalPlan -Dexperiment=fr.inria.exmo.lazylavender.decisiontaking.Experiment ${OPT} -DresultDir=${OUTPUT}

Experimental plan

The independent variables have been varied as follows:

Environment bias coefficient: [0.0, 1.0]
Inverse social bias coefficient: [0.0, 0.3, 0.7, 1.0]
Frequency of adaptation: [0.05, 1.0]

Hypothesis testing

Success rate converges to 1. Improve the average accuracy at the end of the experiment. Agents do not necessarily converge to the same ontology.

Data exploration

Summary of data with adaptation

Environment bias coefficient: [0.0, 1.0]
Inverse social bias coefficient: [0.0, 0.3, 0.7, 1.0]
Frequency of adaptation: [0.05, 1.0]
Environment bias coefficient Inverse social bias coefficient Frequency of adaptation
0.0 1.0 0.0 0.3 0.7 1.0 0.05 1.0
accuracy 1 0.4693 0.4775 0.4967 0.4706 0.4592 0.4670 0.4773 0.4695
1000000 0.3409 0.9081 0.6363 0.6197 0.6514 0.5906 0.6021 0.6469
ssrate 1 0.4000 0.3750 0.3000 0.3500 0.4500 0.4500 0.4000 0.3750
1000000 0.7352 0.8700 0.8533 0.8090 0.7837 0.7644 0.6328 0.9724

Results with adaptation frequency at 0.05

for environment success index weight = 0
coeffInvSoc
0.0000 0.3000 0.7000 1.0000
accuracy 0 0.5306 0.4425 0.4419 0.4788
99999 0.3362 0.3163 0.3494 0.3125
ssrate 0 0.4000 0.4000 0.4000 0.4000
99999 0.6059 0.4823 0.4731 0.4839
for environment success index weight = 1
coeffInvSoc
0.0000 0.3000 0.7000 1.0000
accuracy 0 0.4994 0.4919 0.4906 0.4425
99999 0.8713 0.9250 0.9187 0.7875
ssrate 0 0.2000 0.2000 0.6000 0.6000
99999 0.8469 0.8069 0.7256 0.6377

The success rate does not converge without transmission bias nor with the rarity bias alone.

The accuracy appears to benefit from the rarity bias.

Compare how spread the decision for each rarity bias weight when success bias weight = 1

winDspread includes the agent that just adapted and the agent with which it interacted.

Inverse social bias coefficient
0.0 0.3 0.7 1.0
loss 1000000 2.0000 0.8000 0.8000 1.4000
winDspread 1 10.9370 10.2987 9.3959 9.1095

ANOVA Results when success bias weight = 1:

influence of Inverse social bias coefficient
PR(>F) Significance
ssrate 0.005293 True
accuracy 0.019759 False 






for ssrate
group1 group2 meandiff p-adj lower upper reject
0 0 0.3 -0.0399 0.8610 -0.1903 0.1104 False
1 0 0.7 -0.1213 0.1379 -0.2716 0.0291 False
2 0 1 -0.2092 0.0053 -0.3596 -0.0588 True
3 0.3 0.7 -0.0813 0.4359 -0.2317 0.0691 False
4 0.3 1 -0.1693 0.0248 -0.3196 -0.0189 True
5 0.7 1 -0.0879 0.3695 -0.2383 0.0624 False 
for accuracy
group1 group2 meandiff p-adj lower upper reject
0 0 0.3 0.0537 0.5953 -0.0690 0.1765 False
1 0 0.7 0.0475 0.6738 -0.0753 0.1703 False
2 0 1 -0.0837 0.2465 -0.2065 0.0390 False
3 0.3 0.7 -0.0063 0.9000 -0.1290 0.1165 False
4 0.3 1 -0.1375 0.0256 -0.2603 -0.0147 True
5 0.7 1 -0.1312 0.0341 -0.2540 -0.0085 True