Experiment 20230523-VBCE

Experiment design

Agents adapt ontologies to agree on decision taking. Introducing cultural values (independece, novelty, authority, mastery) that influence which agent adapts in case of interaction failure.

Date: 20230523 (Adriana Luntraru)

Hypotheses: Positive mastery is needed for increasing accuracy. Negative independence causes the success rate to converge faster. Negative novelty increases ontology distance. Positive authority increases accuracy when used with positive mastery.

5 runs; 100000 games

Experimental setting: Agents learn decision trees (transformed into ontologies); get payoffs according to cultural values; adapt by splitting their leaf nodes

Independent variables: ['Independence weight', 'Novelty weight', 'Authority weight', 'Mastery weight']

Dependent variables: ['success rate', 'accuracy', 'ontology distance']

Experiment

Date: 20230523 (Adriana Luntraru)

Lazy lavender hash: bbc1944ad068a5e9e620867c5324f57cab998b31

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

CURRDIR=$(pwd)
OUTPUT=${CURRDIR}/${DIRPREF}
EXP=fr.inria.exmo.lazylavender.decisiontaking.culturalvalues.CulturalValuesExperiment
cd lazylav
# this sample runs ExperimentalPlan. It can be replaced with Monitor if parameters are not varied.
bash scripts/runexp.sh -p ${CURRDIR} -d ${DIRPREF} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.ExperimentalPlan -Dexperiment=${EXP} ${OPT} -DresultDir=${OUTPUT}

Experimental plan

The independent variables have been varied as follows:

Independence weight: [-1.0, 0.0, 1.0]
Novelty weight: [-1.0, 0.0, 1.0]
Authority weight: [-1.0, 0.0, 1.0]
Mastery weight: [-1.0, 0.0, 1.0]

Raw results

Full results are available on zenodo.

DOI DOI 10.5281/zenodo.8124000 10.5281/zenodo.8124000

Data exploration

Use one cultural value index on its own

Only one weight has a non-zero value. Last column corresponds to all weights = 0.

Independence weight Novelty weight Authority weight Mastery weight All weights=0
-1.0 1.0 -1.0 1.0 -1.0 1.0 -1.0 1.0
accuracy 1 0.4556 0.4587 0.4631 0.4456 0.4894 0.4562 0.5019 0.4775 0.5062
20000 0.5269 0.3525 0.3869 0.4344 0.4969 0.4469 0.0694 0.8200 0.5031
40000 0.5231 0.3812 0.3875 0.4313 0.4837 0.4500 0.0212 0.9600 0.5463
60000 0.5319 0.4012 0.3875 0.4219 0.4944 0.4381 0.0125 0.9744 0.5388
80000 0.5375 0.3894 0.3875 0.4319 0.4750 0.4375 0.0125 0.9750 0.5413
100000 0.5375 0.3925 0.3875 0.4237 0.4556 0.4425 0.0125 0.9750 0.5487
ssrate 1 1.0000 0.4000 0.0000 0.2000 0.2000 0.4000 0.4000 0.6000 0.6000
20000 0.7680 0.4447 0.8246 0.5510 0.5393 0.5704 0.5985 0.6712 0.5701
40000 0.8416 0.5030 0.9121 0.6309 0.6104 0.6463 0.7123 0.7898 0.6395
60000 0.8820 0.5482 0.9414 0.6922 0.6671 0.6962 0.7724 0.8569 0.6869
80000 0.9099 0.5799 0.9561 0.7383 0.7152 0.7322 0.8124 0.8926 0.7236
100000 0.9279 0.6054 0.9648 0.7711 0.7520 0.7642 0.8388 0.9141 0.7540
distance 1 0.6244 0.6641 0.6561 0.6604 0.6553 0.6702 0.6670 0.6613 0.6678
20000 0.6168 0.4493 0.7545 0.3654 0.5541 0.5470 0.5643 0.6141 0.5708
40000 0.5829 0.3476 0.7545 0.3256 0.4426 0.4425 0.4872 0.5398 0.4582
60000 0.5797 0.3202 0.7545 0.3244 0.4202 0.4229 0.4623 0.5338 0.4294
80000 0.5797 0.3162 0.7545 0.3244 0.4161 0.4175 0.4568 0.5338 0.4167
100000 0.5797 0.3162 0.7545 0.3244 0.4147 0.4162 0.4531 0.5338 0.4144

Use mastery index weight = 1 with one other index

Mastery weight = 1. Only one other weight has a non-zero value. Last column corresponds to mastery weight = 1 and all other weights = 0.

Independence weight Novelty weight Authority weight Mastery weight=1 alone
-1.0 1.0 -1.0 1.0 -1.0 1.0
accuracy 1 0.4194 0.4569 0.4613 0.4237 0.4319 0.4913 0.4775
20000 0.6244 0.7181 0.4750 0.6338 0.6787 0.7881 0.8200
40000 0.7250 0.8287 0.4750 0.8300 0.8194 0.9106 0.9600
60000 0.7481 0.9144 0.4750 0.8750 0.8725 0.9375 0.9744
80000 0.7537 0.9431 0.4750 0.8750 0.8875 0.9375 0.9750
100000 0.7625 0.9481 0.4750 0.8750 0.8875 0.9375 0.9750
ssrate 1 0.0000 0.2000 0.6000 0.2000 0.0000 0.4000 0.6000
20000 0.6595 0.5567 0.8805 0.6068 0.5976 0.7138 0.6712
40000 0.7665 0.6608 0.9402 0.7177 0.7115 0.7983 0.7898
60000 0.8325 0.7330 0.9602 0.8002 0.7897 0.8596 0.8569
80000 0.8730 0.7877 0.9701 0.8487 0.8388 0.8942 0.8926
100000 0.8981 0.8243 0.9761 0.8790 0.8703 0.9153 0.9141
distance 1 0.6819 0.6380 0.6429 0.6584 0.6471 0.6555 0.6613
20000 0.6054 0.5299 0.7285 0.3891 0.5334 0.6310 0.6141
40000 0.5477 0.4493 0.7285 0.3582 0.4427 0.5433 0.5398
60000 0.5447 0.4338 0.7285 0.3568 0.4364 0.5357 0.5338
80000 0.5439 0.4313 0.7285 0.3568 0.4355 0.5357 0.5338
100000 0.5439 0.4313 0.7285 0.3568 0.4355 0.5357 0.5338

Statistical Analysis

ANOVA and posthoc Tukey HSD tests are performed to determine which independent variables significantly influence which dependent variables.

Four hypotheses are tested:

  • Hypothesis 1: The accuracy does not improve without using a positive success index weight and using a negative success index weight results in the accuracy converging to 0. (supported)
  • Hypothesis 2: A negative independence weight will cause the success rate to converge faster to a higher value. (partially supported)
  • Hypothesis 3: A negative novelty weight will lead to a high final ontology distance. (supported)
  • Hypothesis 4: A positive authority weight will lead to an increase in final accuracy when used together with a positive success weight. (not supported)

Using one cultural value index on its own

Influence of Independence weight:

ANOVA:
  PR(>F) Significance
accuracy 0.113955 False
ssrate 0.000002 True
distance 0.000511 True


Posthoc Tukey for accuracy:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.0112 0.9881 -0.1920 0.2145 False
1 -1 1 -0.1450 0.1801 -0.3482 0.0582 False
2 0 1 -0.1562 0.1424 -0.3595 0.0470 False
Posthoc Tukey for ssrate:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.1739 0.0005 -0.2608 -0.0870 True
1 -1 1 -0.3226 0.0000 -0.4095 -0.2356 True
2 0 1 -0.1487 0.0017 -0.2356 -0.0618 True
Posthoc Tukey for distance:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.1653 0.0129 -0.2940 -0.0365 True
1 -1 1 -0.2635 0.0004 -0.3923 -0.1347 True
2 0 1 -0.0982 0.1462 -0.2270 0.0306 False






Influence of Novelty weight:

ANOVA:
  PR(>F) Significance
accuracy 0.067150 False
ssrate 0.000004 True
distance 0.000000 True


Posthoc Tukey for accuracy:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.1612 0.0683 -0.0116 0.3341 False
1 -1 1 0.0362 0.8436 -0.1366 0.2091 False
2 0 1 -0.1250 0.1728 -0.2978 0.0478 False
Posthoc Tukey for ssrate:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.2108 0.0000 -0.2785 -0.1431 True
1 -1 1 -0.1938 0.0000 -0.2614 -0.1261 True
2 0 1 0.0170 0.7838 -0.0506 0.0847 False
Posthoc Tukey for distance:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.3400 0.0000 -0.4207 -0.2594 True
1 -1 1 -0.4301 0.0000 -0.5108 -0.3494 True
2 0 1 -0.0901 0.0288 -0.1707 -0.0094 True






Influence of Authority weight:

ANOVA:
  PR(>F) Significance
accuracy 0.411979 False
ssrate 0.899969 False
distance 0.997836 False


Posthoc Tukey for accuracy:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.0931 0.5256 -0.1304 0.3167 False
1 -1 1 -0.0131 0.9866 -0.2367 0.2104 False
2 0 1 -0.1062 0.4387 -0.3298 0.1173 False
Posthoc Tukey for ssrate:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.0020 0.9972 -0.0733 0.0773 False
1 -1 1 0.0121 0.9038 -0.0632 0.0874 False
2 0 1 0.0101 0.9319 -0.0652 0.0854 False
Posthoc Tukey for distance:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.0003 0.9999 -0.0766 0.0760 False
1 -1 1 0.0015 0.9986 -0.0748 0.0777 False
2 0 1 0.0018 0.9979 -0.0745 0.0780 False






Influence of Mastery weight:

ANOVA:
  PR(>F) Significance
accuracy 0.000000 True
ssrate 0.000043 True
distance 0.004629 True


Posthoc Tukey for accuracy:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.5362 0.0000 0.3825 0.6900 True
1 -1 1 0.9625 0.0000 0.8088 1.1162 True
2 0 1 0.4262 0.0000 0.2725 0.5800 True
Posthoc Tukey for ssrate:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.0848 0.0064 -0.1440 -0.0256 True
1 -1 1 0.0753 0.0137 0.0161 0.1345 True
2 0 1 0.1601 0.0000 0.1009 0.2193 True
Posthoc Tukey for distance:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.0387 0.4091 -0.1166 0.0392 False
1 -1 1 0.0807 0.0423 0.0028 0.1586 True
2 0 1 0.1193 0.0040 0.0414 0.1972 True






Use mastery index weight = 1 with one other index

Influence of Independence weight:

ANOVA:
  PR(>F) Significance
accuracy 0.000459 True
ssrate 0.008515 True
distance 0.034418 False


Posthoc Tukey for accuracy:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.2125 0.0007 0.1020 0.3230 True
1 -1 1 0.1856 0.0020 0.0751 0.2961 True
2 0 1 -0.0269 0.7965 -0.1374 0.0836 False
Posthoc Tukey for ssrate:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.0160 0.8026 -0.0510 0.0830 False
1 -1 1 -0.0738 0.0309 -0.1408 -0.0068 True
2 0 1 -0.0898 0.0099 -0.1568 -0.0228 True
Posthoc Tukey for distance:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.0101 0.9676 -0.1207 0.1004 False
1 -1 1 -0.1126 0.0458 -0.2232 -0.0021 True
2 0 1 -0.1025 0.0702 -0.2130 0.0081 False






Influence of Novelty weight:

ANOVA:
  PR(>F) Significance
accuracy 0.000001 True
ssrate 0.000075 True
distance 0.000000 True


Posthoc Tukey for accuracy:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.5000 0.0000 0.3694 0.6306 True
1 -1 1 0.4000 0.0000 0.2694 0.5306 True
2 0 1 -0.1000 0.1443 -0.2306 0.0306 False
Posthoc Tukey for ssrate:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.0620 0.0028 -0.1005 -0.0235 True
1 -1 1 -0.0971 0.0001 -0.1356 -0.0586 True
2 0 1 -0.0351 0.0748 -0.0736 0.0033 False
Posthoc Tukey for distance:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 -0.1948 0.0000 -0.2682 -0.1213 True
1 -1 1 -0.3717 0.0000 -0.4452 -0.2982 True
2 0 1 -0.1769 0.0001 -0.2504 -0.1035 True






Influence of Authority weight:

ANOVA:
  PR(>F) Significance
accuracy 0.090169 False
ssrate 0.046613 False
distance 0.005096 True


Posthoc Tukey for accuracy:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.0875 0.0763 -0.0088 0.1838 False
1 -1 1 0.0500 0.3786 -0.0463 0.1463 False
2 0 1 -0.0375 0.5676 -0.1338 0.0588 False
Posthoc Tukey for ssrate:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.0439 0.0775 -0.0046 0.0923 False
1 -1 1 0.0451 0.0690 -0.0034 0.0935 False
2 0 1 0.0012 0.9975 -0.0472 0.0497 False
Posthoc Tukey for distance:
group1 group2 meandiff p-adj lower upper reject
0 -1 0 0.0983 0.0107 0.0240 0.1726 True
1 -1 1 0.1002 0.0095 0.0259 0.1745 True
2 0 1 0.0019 0.9974 -0.0724 0.0762 False