Experiment 20210224-DOLA¶

Experiment design¶

Agent adapt ontologies to agree on decision taking

Date: 20210224 (Yasser Bourahla)

5 runs; 100000 games

Hypothesis: ['hypotheses of 20200623-DOLA are still verified for different training algorithms and starting with random ontologies', '']

Variation of: 20200623-DOLA

Decision tree learning algorithms: RAND,ID3,C45,CART

Variables¶

independent variables: ['numberOfClasses', 'ratio', 'numberOfFeatures', 'trainer']

dependent variables: ['ssrate', 'accuracy', 'distance']

Experiment¶

Date: 20210224 (Yasser Bourahla)

LazyLavender hash: 37dffefd52354174aac791c6513cae2a564428f7

Link to lazylavender

Parameter file: params.sh

Executed command (script.sh):

BEWARE: REPRODUCING THE ANALYSIS MAY TAKE A CONSIDERABLE AMOUNT OF TIME.

#!/bin/bash

. params.sh

CURRDIR=$(pwd)
OUTPUT=${CURRDIR}/${DIRPREF}
# cd ${LLPATH}
cd lazylav
# this sample runs ExperimentalPlan. It can be replaced with Monitor if parameters are not varied.
bash scripts/runexp.sh -p ${CURRDIR} -d ${DIRPREF} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.ExperimentalPlan -Dexperiment=fr.inria.exmo.lazylavender.decisiontaking.Experiment ${OPT} -DresultDir=${OUTPUT}

Experimental plan¶

The independent variables have been varied as follows:

number of classes: [2, 3, 4]

training ratio: [0.1, 0.3, 0.5]

number of features: [3, 4, 5]

ontology initialization: ['C45', 'CART', 'ID3', 'RAND']

		number of classes			training ratio			number of features			ontology initialization
		2	3	4	0.1	0.3	0.5	3	4	5	C45	CART	ID3	RAND
ssrate	1	0.55	0.43	0.38	0.39	0.46	0.51	0.44	0.49	0.43	0.50	0.50	0.50	0.32
ssrate	1000000	0.94	0.90	0.88	0.92	0.89	0.91	0.97	0.92	0.82	0.92	0.91	0.92	0.87
accuracy	1	0.61	0.49	0.42	0.42	0.50	0.60	0.51	0.51	0.50	0.56	0.55	0.56	0.36
accuracy	1000000	0.89	0.84	0.80	0.73	0.88	0.92	0.86	0.85	0.82	0.85	0.87	0.86	0.79
distance	1	0.64	0.66	0.66	0.50	0.73	0.73	0.51	0.66	0.79	0.61	0.65	0.61	0.74
distance	1000000	0.51	0.44	0.40	0.43	0.46	0.45	0.36	0.47	0.51	0.45	0.44	0.44	0.46

Data exploration¶

Results without random ontology initialisation¶

Hypothesis 1:¶

hypothesis 1 verified

Hypothesis 2:¶

There is a significant difference in the accuracy between the start and the end of the experiment.
Paired t-test results: t=-55.39 and p<0.01.
Hypothesis 2 verified.

Hypothesis 3:¶

the percentage of final none zero distances to all distances is : 98.52 %
hypothesis 2 verified.

Note here, the percentage of none zero distances is 98.52% (higher than previous 90.78%) because the experiments are carried with 20 agents only.

Comparison between learning algorithms:¶

ANOVA results (without random ontology initialisation):

lattice of parameter influence on ssrate:

lattice of parameter influence on accuracy:

lattice of parameter influence on distance:

Confidence Intervals of accuracy difference:

CART - C45 [ -0.0246 , 0.0585 ]
ID3 - C45 [ -0.0318 , 0.0513 ]
ID3 - CART [ -0.0487 , 0.0344 ]

Confidence intervals are all within: [-0.0487, 0.0585]. There is no significant difference between the learning algorithms.

interaction of numberOfClasses with trainer

interaction of ratio with trainer

interaction of numberOfFeatures with trainer

Results with random ontology initialisation only¶